Sep 16 19:05:15.670564 ip-10-0-128-148 systemd[1]: Starting Kubernetes Kubelet... Sep 16 19:05:16.068195 ip-10-0-128-148 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:05:16.068195 ip-10-0-128-148 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 19:05:16.068195 ip-10-0-128-148 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:05:16.068195 ip-10-0-128-148 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 19:05:16.068195 ip-10-0-128-148 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 19:05:16.069742 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.069657 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073875 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073896 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073900 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073904 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073906 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.073904 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073912 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073917 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073920 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073923 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073926 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073929 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073931 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073934 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073936 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073939 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073942 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073945 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073947 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073950 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073952 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073955 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073957 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073960 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073962 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073965 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.074130 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073968 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073970 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073973 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073975 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073977 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073986 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073989 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073992 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073995 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.073998 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074000 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074002 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074005 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074007 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074010 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074013 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074017 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074020 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074022 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074025 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.074627 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074028 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074030 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074032 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074035 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074038 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074040 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074043 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074045 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074048 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074050 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074052 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074055 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074059 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074062 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074065 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074069 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074073 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074076 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074080 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.075131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074083 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074087 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074090 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074093 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074096 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074098 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074101 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074103 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074107 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074109 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074112 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074115 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074117 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074120 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074122 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074125 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074127 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074130 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074133 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074136 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.075605 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074139 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074142 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074144 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074147 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074149 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074152 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074155 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074159 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074163 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074167 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074171 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074176 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074179 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074183 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074187 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074912 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074919 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074922 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074925 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074929 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.076203 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074932 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074935 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074937 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074940 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074943 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074945 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074948 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074951 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074954 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074956 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074959 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074961 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074965 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074967 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074970 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074973 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074975 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074978 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074981 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074983 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.076695 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074986 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074988 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074991 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074995 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.074999 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075003 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075006 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075009 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075011 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075014 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075017 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075019 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075022 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075025 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075027 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075030 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075032 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075034 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075037 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075040 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.077191 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075042 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075045 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075047 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075050 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075052 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075055 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075058 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075060 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075063 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075065 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075068 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075070 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075072 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075075 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075080 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075084 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075086 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075089 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075093 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.077689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075096 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075099 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075102 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075105 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075109 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075111 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075114 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075116 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075119 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075121 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075124 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075126 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075129 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075131 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075134 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075136 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075139 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075141 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075144 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075148 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.078184 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075150 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075153 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075156 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075159 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075161 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075164 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075166 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075169 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075171 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075174 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075176 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075179 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075181 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075183 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075186 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075574 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075579 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075582 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075585 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075587 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.078687 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075590 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075593 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075595 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075598 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075602 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075605 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075608 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075611 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075613 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075616 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075619 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075623 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075625 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075628 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075633 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075636 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075640 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075644 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075648 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.079180 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075651 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075654 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075657 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075660 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075663 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075665 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075668 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075670 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075674 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075677 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075679 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075682 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075684 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075687 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075689 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075692 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075694 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075697 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075699 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075702 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.079693 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075704 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075707 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075709 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075712 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075715 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075718 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075721 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075724 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075727 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075730 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075732 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075735 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075737 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075739 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075742 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075746 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075749 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075755 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075758 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075760 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.080183 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075763 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075766 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075768 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075771 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075773 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075776 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075778 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075781 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075783 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075785 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075788 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075790 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075792 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075795 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075797 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075800 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075802 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075805 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075808 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.080689 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075812 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075815 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075818 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075821 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075823 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075826 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075828 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075831 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075833 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075835 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075838 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075841 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075843 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075846 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075848 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.075850 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075923 2565 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075961 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075967 2565 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075971 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075975 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 19:05:16.081179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075978 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075982 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075986 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075989 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075992 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075996 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.075999 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076002 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076006 2565 flags.go:64] FLAG: --cgroup-root="" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076009 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076012 2565 flags.go:64] FLAG: --client-ca-file="" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076016 2565 flags.go:64] FLAG: --cloud-provider="external" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076019 2565 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076024 2565 flags.go:64] FLAG: --cluster-domain="" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076027 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076029 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076032 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076035 2565 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076039 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076042 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076045 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076048 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076051 2565 flags.go:64] FLAG: --contention-profiling="false" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076054 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 19:05:16.081713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076057 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076060 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076063 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076067 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076070 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076073 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076075 2565 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076078 2565 flags.go:64] FLAG: --enable-server="true" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076081 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076085 2565 flags.go:64] FLAG: --event-burst="100" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076088 2565 flags.go:64] FLAG: --event-qps="50" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076091 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076094 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076096 2565 flags.go:64] FLAG: --eviction-hard="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076100 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076103 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076107 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076110 2565 flags.go:64] FLAG: --eviction-soft="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076113 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076117 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076120 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076123 2565 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076126 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076129 2565 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076132 2565 flags.go:64] FLAG: --feature-gates="" Sep 16 19:05:16.082338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076135 2565 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076138 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076141 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076144 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076147 2565 flags.go:64] FLAG: --healthz-port="10248" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076150 2565 flags.go:64] FLAG: --help="false" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076152 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076156 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076158 2565 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076161 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076164 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076168 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076170 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076173 2565 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076176 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076179 2565 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076181 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076184 2565 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076187 2565 flags.go:64] FLAG: --kube-reserved="" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076190 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076192 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076195 2565 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076198 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076202 2565 flags.go:64] FLAG: --lock-file="" Sep 16 19:05:16.083009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076204 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076207 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076212 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076217 2565 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076220 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076223 2565 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076226 2565 flags.go:64] FLAG: --logging-format="text" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076229 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076232 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076234 2565 flags.go:64] FLAG: --manifest-url="" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076237 2565 flags.go:64] FLAG: --manifest-url-header="" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076241 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076244 2565 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076248 2565 flags.go:64] FLAG: --max-pods="110" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076251 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076254 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076256 2565 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076259 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076262 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076265 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076267 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076274 2565 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076291 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076294 2565 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 19:05:16.083614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076297 2565 flags.go:64] FLAG: --pod-cidr="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076300 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076307 2565 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076310 2565 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076313 2565 flags.go:64] FLAG: --pods-per-core="0" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076315 2565 flags.go:64] FLAG: --port="10250" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076318 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076322 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-036f405379a20701a" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076326 2565 flags.go:64] FLAG: --qos-reserved="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076328 2565 flags.go:64] FLAG: --read-only-port="10255" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076331 2565 flags.go:64] FLAG: --register-node="true" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076335 2565 flags.go:64] FLAG: --register-with-taints="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076338 2565 flags.go:64] FLAG: --registry-burst="10" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076341 2565 flags.go:64] FLAG: --registry-qps="5" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076344 2565 flags.go:64] FLAG: --reserved-cpus="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076347 2565 flags.go:64] FLAG: --reserved-memory="" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076350 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076353 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076356 2565 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076359 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076362 2565 flags.go:64] FLAG: --runonce="false" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076365 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076367 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076370 2565 flags.go:64] FLAG: --seccomp-default="false" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076373 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 19:05:16.084197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076376 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076378 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076381 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076384 2565 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076387 2565 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076390 2565 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076392 2565 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076395 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076398 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076401 2565 flags.go:64] FLAG: --system-cgroups="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076403 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076408 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076411 2565 flags.go:64] FLAG: --tls-cert-file="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076414 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076419 2565 flags.go:64] FLAG: --tls-min-version="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076422 2565 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076424 2565 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076427 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076430 2565 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076433 2565 flags.go:64] FLAG: --v="2" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076437 2565 flags.go:64] FLAG: --version="false" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076441 2565 flags.go:64] FLAG: --vmodule="" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076445 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076448 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 19:05:16.084862 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076558 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076562 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076567 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076570 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076573 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076576 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076579 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076582 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076585 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076588 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076590 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076593 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076596 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076598 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076601 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076604 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076606 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076609 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076611 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076614 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.085448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076616 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076619 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076623 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076625 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076628 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076631 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076633 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076636 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076638 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076641 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076644 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076647 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076650 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076652 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076654 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076657 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076659 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076662 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076664 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076667 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.085949 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076669 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076672 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076675 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076677 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076680 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076684 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076687 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076690 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076693 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076696 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076699 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076702 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076705 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076708 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076713 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076716 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076718 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076721 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076724 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076726 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.086523 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076729 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076731 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076734 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076737 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076739 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076741 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076744 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076746 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076749 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076751 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076754 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076756 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076760 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076763 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076767 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076770 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076773 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076776 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076779 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.087010 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076782 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076784 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076787 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076791 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076793 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076796 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076799 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076802 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076805 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076807 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076810 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076812 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076815 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076818 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076820 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076823 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076825 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076828 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076831 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.087484 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.076833 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.076838 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.084078 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.084093 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.084753 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.084760 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084817 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084824 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084828 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084830 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084833 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084836 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084839 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084841 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.088043 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084844 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084847 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084849 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084852 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084855 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084859 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084862 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084865 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084868 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084871 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084874 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084877 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084880 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084882 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084885 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084887 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084890 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084892 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084895 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084897 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.088421 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084899 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084903 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084906 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084909 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084911 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084914 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084916 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084919 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084922 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084925 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084927 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084930 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084932 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084935 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084937 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084940 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084942 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084945 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084948 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084950 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.088931 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084952 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084955 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084957 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084960 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084963 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084965 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084968 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084970 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084973 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084975 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084978 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084980 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084983 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084986 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084989 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084992 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084994 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084997 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.084999 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085002 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.089432 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085004 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085007 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085009 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085012 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085014 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085017 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085019 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085022 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085024 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085027 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085030 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085032 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085035 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085039 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085042 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085045 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085048 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085051 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085054 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.089925 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085056 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085059 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085061 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085064 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085066 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085069 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085072 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085075 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085079 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085081 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085084 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085087 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.085091 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085200 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 19:05:16.090458 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085205 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085207 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085210 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085213 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085215 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085218 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085222 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085225 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085228 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085231 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085234 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085237 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085240 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085243 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085245 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085248 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085250 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085253 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085255 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 19:05:16.090856 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085258 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085260 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085263 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085265 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085268 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085271 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085274 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085288 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085292 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085295 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085298 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085300 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085303 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085305 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085308 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085311 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085313 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085316 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085318 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085321 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 19:05:16.091339 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085323 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085326 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085329 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085332 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085335 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085338 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085340 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085343 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085346 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085348 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085350 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085353 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085355 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085357 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085360 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085362 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085365 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085368 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085371 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 19:05:16.091833 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085373 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085376 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085379 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085381 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085384 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085386 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085389 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085391 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085394 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085396 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085399 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085401 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085404 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085406 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085409 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085411 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085414 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085416 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085419 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085421 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 19:05:16.092297 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085424 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085426 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085429 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085431 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085434 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085436 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085439 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085441 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085443 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085447 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085451 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085455 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085458 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085461 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085464 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085473 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085476 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085479 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085481 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 19:05:16.092779 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:16.085484 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.085489 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.085594 2565 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.087595 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.088487 2565 server.go:1019] "Starting client certificate rotation" Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.088583 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.088651 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 19:05:16.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.090646 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 19:05:16.095824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.095801 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 19:05:16.106203 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.106183 2565 log.go:25] "Validated CRI v1 runtime API" Sep 16 19:05:16.109446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.109429 2565 log.go:25] "Validated CRI v1 image API" Sep 16 19:05:16.111930 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.111915 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 19:05:16.113922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.113888 2565 fs.go:135] Filesystem UUIDs: map[78e368fa-8a59-4c4b-93f6-514e3206d68c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b622ccf2-1054-423d-843b-3a24b24f5ce1:/dev/nvme0n1p3] Sep 16 19:05:16.113995 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.113915 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 19:05:16.119215 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.119105 2565 manager.go:216] Machine: {Timestamp:2026-09-16 19:05:16.117887478 +0000 UTC m=+0.323843222 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099636 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5ac3dcabafa5e920ed501247194a SystemUUID:ec2a5ac3-dcab-afa5-e920-ed501247194a BootID:7b73f045-fc2d-447e-bc7d-412a23ed2dcd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:74:c4:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:74:c4:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:1b:44:ad:1c:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 19:05:16.119215 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.119207 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 19:05:16.119356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.119299 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 19:05:16.120785 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.120764 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 19:05:16.122188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.122152 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 19:05:16.122347 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.122187 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 19:05:16.124086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.123840 2565 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 19:05:16.124086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.123857 2565 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 19:05:16.124086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.123865 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 19:05:16.125419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.124627 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 19:05:16.125932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.125723 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 19:05:16.128361 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128350 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 16 19:05:16.128484 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128474 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 19:05:16.128529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128504 2565 kubelet.go:486] "Attempting to sync node with API server" Sep 16 19:05:16.128529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128514 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 19:05:16.128529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128527 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 19:05:16.128628 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128537 2565 kubelet.go:398] "Adding apiserver pod source" Sep 16 19:05:16.128628 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128546 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 19:05:16.128994 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.128982 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 19:05:16.129039 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.129007 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 19:05:16.131053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.131029 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 19:05:16.134957 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.134940 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 19:05:16.135031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.134973 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 19:05:16.136590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136575 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136594 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136601 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136606 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136618 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136624 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 19:05:16.136638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136637 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 19:05:16.136793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136642 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 19:05:16.136793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136649 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 19:05:16.136793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136655 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 19:05:16.136793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136669 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 19:05:16.136793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.136674 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 19:05:16.138566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.138552 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 19:05:16.138566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.138567 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 19:05:16.140668 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.140642 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 19:05:16.140668 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.140642 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 19:05:16.142343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.142331 2565 server.go:1268] "Started kubelet" Sep 16 19:05:16.142458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.142409 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 19:05:16.142518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.142477 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 19:05:16.142518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.142480 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 19:05:16.143124 ip-10-0-128-148 systemd[1]: Started Kubernetes Kubelet. Sep 16 19:05:16.143542 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.143525 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 19:05:16.144375 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.144358 2565 server.go:310] "Adding debug handlers to kubelet server" Sep 16 19:05:16.146920 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.146903 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 19:05:16.149414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.149395 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 19:05:16.150760 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.150742 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 19:05:16.150890 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.150870 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.151330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151312 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 19:05:16.151330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151331 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 19:05:16.151487 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151330 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 19:05:16.151487 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151438 2565 reconstruct.go:97] "Volume reconstruction finished" Sep 16 19:05:16.151487 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151446 2565 reconciler.go:29] "Reconciler: start to sync state" Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151838 2565 factory.go:153] Registering CRI-O factory Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151853 2565 factory.go:223] Registration of the crio container factory successfully Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151903 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151912 2565 factory.go:55] Registering systemd factory Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151919 2565 factory.go:223] Registration of the systemd container factory successfully Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151935 2565 factory.go:103] Registering Raw factory Sep 16 19:05:16.151972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.151955 2565 manager.go:1196] Started watching for new ooms in manager Sep 16 19:05:16.152372 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.152359 2565 manager.go:349] Starting recovery of all containers Sep 16 19:05:16.154349 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.154325 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 19:05:16.154495 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.154448 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 19:05:16.154594 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.154571 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 19:05:16.155118 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.154219 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-148.ec2.internal.18d5e28b98d8a4cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-148.ec2.internal,UID:ip-10-0-128-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-148.ec2.internal,},FirstTimestamp:2026-09-16 19:05:16.142306507 +0000 UTC m=+0.348262252,LastTimestamp:2026-09-16 19:05:16.142306507 +0000 UTC m=+0.348262252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-148.ec2.internal,}" Sep 16 19:05:16.155848 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.155828 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5qsw" Sep 16 19:05:16.164596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.164562 2565 manager.go:354] Recovery completed Sep 16 19:05:16.165016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.164994 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5qsw" Sep 16 19:05:16.169116 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.169101 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.171104 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171087 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.171192 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171120 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.171192 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171136 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.171747 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171733 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.171838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171732 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 19:05:16.171838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171791 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 19:05:16.171838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.171812 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 16 19:05:16.172404 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172385 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.172469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172415 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.172469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172427 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.172811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172798 2565 policy_none.go:49] "None policy: Start" Sep 16 19:05:16.172847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172815 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 19:05:16.172847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.172825 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 19:05:16.173699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.173688 2565 policy_none.go:47] "Start" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.210800 2565 manager.go:337] "Starting Device Plugin manager" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.210851 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.210865 2565 server.go:87] "Starting device plugin registration server" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.211156 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.211168 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.211323 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.211441 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.211473 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.211976 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 19:05:16.213787 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.212016 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.245079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.245041 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 19:05:16.246505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.246489 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 19:05:16.246603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.246543 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 19:05:16.246603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.246571 2565 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 19:05:16.246704 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.246614 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 19:05:16.249118 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.249098 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:05:16.311887 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.311851 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.312886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.312871 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.312953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.312902 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.312953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.312914 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.312953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.312934 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.324913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.324867 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.324913 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.324890 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-148.ec2.internal\": node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.347492 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.347323 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"] Sep 16 19:05:16.347580 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.347542 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.348396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.348379 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.348484 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.348408 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.348484 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.348418 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.349627 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.349616 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.349758 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.349745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.349795 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.349775 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.350274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350250 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.350274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350275 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.350424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350253 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.350424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350303 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.350424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350319 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.350424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.350330 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.351371 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.351355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.351445 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.351387 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 19:05:16.351598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.351581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.351650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.351606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.351650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.351621 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.352070 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.352057 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory" Sep 16 19:05:16.352133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.352082 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 19:05:16.352133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.352097 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID" Sep 16 19:05:16.355360 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.355344 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.374051 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.374026 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-148.ec2.internal\" not found" node="ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.378413 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.378393 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-148.ec2.internal\" not found" node="ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.452604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.452485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.455422 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.455403 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.556100 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.556067 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.656422 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.656359 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.676530 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.676505 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.681543 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:16.681525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:16.757342 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.757303 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.857814 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.857778 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:16.958407 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:16.958324 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.058991 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.058960 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.088141 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.088112 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 19:05:17.088630 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.088263 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 19:05:17.157845 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:17.157805 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a59d530e718667cce688f2a3ad43aaa.slice/crio-8c0e8fe17e3cdf989423d6175e64932216bfd6b38fe2a2ed355967a19e3e048b WatchSource:0}: Error finding container 8c0e8fe17e3cdf989423d6175e64932216bfd6b38fe2a2ed355967a19e3e048b: Status 404 returned error can't find the container with id 8c0e8fe17e3cdf989423d6175e64932216bfd6b38fe2a2ed355967a19e3e048b Sep 16 19:05:17.158131 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:17.158117 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1353260c7ace9243a51756424e4b1aa.slice/crio-eb471773a33e28c63d3ea2635344eaa379234f22dc0fdaeb1129263d6c5da675 WatchSource:0}: Error finding container eb471773a33e28c63d3ea2635344eaa379234f22dc0fdaeb1129263d6c5da675: Status 404 returned error can't find the container with id eb471773a33e28c63d3ea2635344eaa379234f22dc0fdaeb1129263d6c5da675 Sep 16 19:05:17.159197 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.159178 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.161911 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.161898 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:05:17.168447 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.168417 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 19:00:16 +0000 UTC" deadline="2028-05-16 18:06:43.95472151 +0000 UTC" Sep 16 19:05:17.168447 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.168445 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14591h1m26.786278918s" Sep 16 19:05:17.250216 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.250106 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" event={"ID":"f1353260c7ace9243a51756424e4b1aa","Type":"ContainerStarted","Data":"eb471773a33e28c63d3ea2635344eaa379234f22dc0fdaeb1129263d6c5da675"} Sep 16 19:05:17.251200 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.251179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerStarted","Data":"8c0e8fe17e3cdf989423d6175e64932216bfd6b38fe2a2ed355967a19e3e048b"} Sep 16 19:05:17.259534 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.259515 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.360002 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.359970 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.373566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.373547 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:05:17.460934 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.460899 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.519245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.519170 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:05:17.562050 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:17.562005 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found" Sep 16 19:05:17.594507 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.594474 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:05:17.651165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.650916 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" Sep 16 19:05:17.663633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.663605 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 19:05:17.664632 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.664612 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" Sep 16 19:05:17.674565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:17.674546 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 19:05:18.129755 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.129718 2565 apiserver.go:52] "Watching apiserver" Sep 16 19:05:18.138258 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.138224 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 19:05:18.138754 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.138726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cpgbn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp","openshift-dns/node-resolver-9fwsh","openshift-image-registry/node-ca-mj67g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal","openshift-multus/multus-additional-cni-plugins-fp9xj","openshift-network-diagnostics/network-check-target-kg6ff","openshift-network-operator/iptables-alerter-9bc4k","kube-system/konnectivity-agent-vkrhj","kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nb9ft","openshift-multus/multus-czckf","openshift-multus/network-metrics-daemon-vnwn8","openshift-ovn-kubernetes/ovnkube-node-hkdmc"] Sep 16 19:05:18.140408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.140385 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:18.140517 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.140478 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:18.141732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.141705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.142221 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.142190 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 19:05:18.142955 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.142929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.144191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.144158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.144850 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.144823 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t45s\"" Sep 16 19:05:18.144988 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.144966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.145194 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.144828 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.145639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.145614 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.145766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.145639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.145910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.145889 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.146153 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.146132 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6h2vz\"" Sep 16 19:05:18.146436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.146420 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.147029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.147007 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.147269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.147249 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.147601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.147581 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 19:05:18.148190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.148171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.150071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.149617 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 19:05:18.150071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.149693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.150071 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.149781 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:18.150718 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.150699 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n7kf9\"" Sep 16 19:05:18.150820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.150778 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.150926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.150899 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 19:05:18.151100 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.151206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151189 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 19:05:18.151437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vwlzg\"" Sep 16 19:05:18.151522 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151481 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 19:05:18.151596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151581 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 19:05:18.151648 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.151727 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 19:05:18.151997 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.151979 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 19:05:18.152102 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.152085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.152166 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.152115 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96mf5\"" Sep 16 19:05:18.152832 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.152792 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 19:05:18.153025 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.153007 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxsz9\"" Sep 16 19:05:18.153229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.153210 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 19:05:18.153790 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.153388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.153790 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.153397 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 19:05:18.153790 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.153771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157116 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157134 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157218 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157239 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157262 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmx5s\"" Sep 16 19:05:18.157425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157432 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r7twt\"" Sep 16 19:05:18.157947 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.157925 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 19:05:18.159139 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.159114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.159236 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.159188 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:18.160898 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.160874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czckf" Sep 16 19:05:18.163485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jjnnw\"" Sep 16 19:05:18.163725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbrr\" (UniqueName: \"kubernetes.io/projected/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-kube-api-access-pvbrr\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.163820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.163820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-kubernetes\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.163820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.163820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-host\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-konnectivity-ca\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163853 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-modprobe-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-var-lib-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-bin\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.163982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-var-lib-kubelet\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-kubelet\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-config\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbvs\" (UniqueName: \"kubernetes.io/projected/297542fb-6cc8-409f-a483-4749340921ef-kube-api-access-dcbvs\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-netns\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovn-node-metrics-cert\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhh5r\" (UniqueName: \"kubernetes.io/projected/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-kube-api-access-bhh5r\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-etc-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-script-lib\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb8z7\" (UniqueName: \"kubernetes.io/projected/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-kube-api-access-jb8z7\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-device-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/297542fb-6cc8-409f-a483-4749340921ef-serviceca\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-conf\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.164615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-socket-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-sys-fs\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-host-slash\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzm24\" (UniqueName: \"kubernetes.io/projected/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-kube-api-access-dzm24\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-netd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjxl\" (UniqueName: \"kubernetes.io/projected/b547de82-20e4-4a98-b580-70f04676b620-kube-api-access-6xjxl\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-node-log\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-run\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.164977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-tmp\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-ovn\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-iptables-alerter-script\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqhn\" (UniqueName: \"kubernetes.io/projected/cf5a33ab-faac-469c-8a6d-f995291ddffc-kube-api-access-8zqhn\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysconfig\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-systemd\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-tuned\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxs4\" (UniqueName: \"kubernetes.io/projected/d71032cf-e13b-454c-9b19-9cd8e1ade547-kube-api-access-blxs4\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cnibin\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-sys\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-log-socket\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/297542fb-6cc8-409f-a483-4749340921ef-host\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-slash\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-registration-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-systemd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-env-overrides\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b547de82-20e4-4a98-b580-70f04676b620-hosts-file\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-lib-modules\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.165964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-systemd-units\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.166685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-agent-certs\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.166685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b547de82-20e4-4a98-b580-70f04676b620-tmp-dir\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.166685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.165716 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 19:05:18.192497 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.192469 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn9c6" Sep 16 19:05:18.201363 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.201340 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn9c6" Sep 16 19:05:18.252561 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.252530 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 19:05:18.265871 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.265838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.265882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-device-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.265909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/297542fb-6cc8-409f-a483-4749340921ef-serviceca\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.266043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.265990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-device-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.265995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.266293 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-conf\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.266293 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.266293 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.266127 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:18.266293 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-socket-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266293 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.266210 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:18.766172357 +0000 UTC m=+2.972128104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-sys-fs\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-conf\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-host-slash\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzm24\" (UniqueName: \"kubernetes.io/projected/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-kube-api-access-dzm24\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266404 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-socket-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-sys-fs\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-k8s-cni-cncf-io\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-host-slash\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266489 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/297542fb-6cc8-409f-a483-4749340921ef-serviceca\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.266569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-netd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-netd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjxl\" (UniqueName: \"kubernetes.io/projected/b547de82-20e4-4a98-b580-70f04676b620-kube-api-access-6xjxl\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-system-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-netns\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcdb\" (UniqueName: \"kubernetes.io/projected/d27b7496-86c5-4818-89b4-8d50a41259bc-kube-api-access-pwcdb\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-node-log\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-run\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-tmp\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.266986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-node-log\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.266993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-ovn\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-iptables-alerter-script\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-kubelet\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqhn\" (UniqueName: \"kubernetes.io/projected/cf5a33ab-faac-469c-8a6d-f995291ddffc-kube-api-access-8zqhn\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysconfig\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-systemd\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-run\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-tuned\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxs4\" (UniqueName: \"kubernetes.io/projected/d71032cf-e13b-454c-9b19-9cd8e1ade547-kube-api-access-blxs4\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cnibin\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-dbus\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-etc-kubernetes\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-sys\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-log-socket\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/297542fb-6cc8-409f-a483-4749340921ef-host\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-socket-dir-parent\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.267535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-slash\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-registration-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-systemd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267354 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-systemd\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cnibin\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-sys\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-log-socket\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267592 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/297542fb-6cc8-409f-a483-4749340921ef-host\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-slash\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-registration-dir\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-systemd\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysconfig\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-conf-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.268490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-env-overrides\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.267026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-run-ovn\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b547de82-20e4-4a98-b580-70f04676b620-hosts-file\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-kubelet-config\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-bin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-hostroot\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-lib-modules\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-systemd-units\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-agent-certs\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b547de82-20e4-4a98-b580-70f04676b620-tmp-dir\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbrr\" (UniqueName: \"kubernetes.io/projected/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-kube-api-access-pvbrr\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-daemon-config\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-lib-modules\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-multus-certs\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-iptables-alerter-script\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-kubernetes\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.269437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b547de82-20e4-4a98-b580-70f04676b620-hosts-file\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-systemd-units\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-host\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-konnectivity-ca\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf5a33ab-faac-469c-8a6d-f995291ddffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-modprobe-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-var-lib-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-bin\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-cnibin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-sysctl-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-cni-binary-copy\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b547de82-20e4-4a98-b580-70f04676b620-tmp-dir\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-var-lib-kubelet\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-kubernetes\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-kubelet\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-env-overrides\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-config\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.270270 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268806 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-kubelet\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbvs\" (UniqueName: \"kubernetes.io/projected/297542fb-6cc8-409f-a483-4749340921ef-kube-api-access-dcbvs\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-netns\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-host\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.268947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovn-node-metrics-cert\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-cni-bin\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269145 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-modprobe-d\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-var-lib-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269239 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-netns\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-konnectivity-ca\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-host-run-ovn-kubernetes\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-var-lib-kubelet\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269422 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-system-cni-dir\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-multus\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.271005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-config\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271839 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhh5r\" (UniqueName: \"kubernetes.io/projected/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-kube-api-access-bhh5r\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.271839 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-etc-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271839 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-script-lib\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.271839 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb8z7\" (UniqueName: \"kubernetes.io/projected/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-kube-api-access-jb8z7\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.271839 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.269801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.272421 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.272388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovnkube-script-lib\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.272589 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.272566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.272673 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.272624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d71032cf-e13b-454c-9b19-9cd8e1ade547-etc-openvswitch\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.272828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.272811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-tmp\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.273330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.273306 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-etc-tuned\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.273588 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.273565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d71032cf-e13b-454c-9b19-9cd8e1ade547-ovn-node-metrics-cert\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.278867 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.278842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57f5c18c-6c8b-4cc8-b36f-67dba09488e7-agent-certs\") pod \"konnectivity-agent-vkrhj\" (UID: \"57f5c18c-6c8b-4cc8-b36f-67dba09488e7\") " pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.280551 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.280214 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:18.280551 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.280238 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:18.280551 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.280253 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:18.280551 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.280324 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:18.780304913 +0000 UTC m=+2.986260665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:18.283551 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.283517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbrr\" (UniqueName: \"kubernetes.io/projected/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-kube-api-access-pvbrr\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.283660 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.283614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzm24\" (UniqueName: \"kubernetes.io/projected/cb895f80-eded-4f74-a5f9-bfeeec6bc53e-kube-api-access-dzm24\") pod \"iptables-alerter-9bc4k\" (UID: \"cb895f80-eded-4f74-a5f9-bfeeec6bc53e\") " pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.283731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.283673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjxl\" (UniqueName: \"kubernetes.io/projected/b547de82-20e4-4a98-b580-70f04676b620-kube-api-access-6xjxl\") pod \"node-resolver-9fwsh\" (UID: \"b547de82-20e4-4a98-b580-70f04676b620\") " pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.283731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.283688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhh5r\" (UniqueName: \"kubernetes.io/projected/8bd60af3-1dd1-4bb8-a22c-c821d2c69412-kube-api-access-bhh5r\") pod \"tuned-nb9ft\" (UID: \"8bd60af3-1dd1-4bb8-a22c-c821d2c69412\") " pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.283844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.283695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxs4\" (UniqueName: \"kubernetes.io/projected/d71032cf-e13b-454c-9b19-9cd8e1ade547-kube-api-access-blxs4\") pod \"ovnkube-node-hkdmc\" (UID: \"d71032cf-e13b-454c-9b19-9cd8e1ade547\") " pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.284993 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.284867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb8z7\" (UniqueName: \"kubernetes.io/projected/5da48f5d-46b1-4fe4-b6e8-3aa0640f74de-kube-api-access-jb8z7\") pod \"multus-additional-cni-plugins-fp9xj\" (UID: \"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de\") " pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.285563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.285538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqhn\" (UniqueName: \"kubernetes.io/projected/cf5a33ab-faac-469c-8a6d-f995291ddffc-kube-api-access-8zqhn\") pod \"aws-ebs-csi-driver-node-zdzgp\" (UID: \"cf5a33ab-faac-469c-8a6d-f995291ddffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.289085 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.289064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbvs\" (UniqueName: \"kubernetes.io/projected/297542fb-6cc8-409f-a483-4749340921ef-kube-api-access-dcbvs\") pod \"node-ca-mj67g\" (UID: \"297542fb-6cc8-409f-a483-4749340921ef\") " pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.370399 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370399 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-k8s-cni-cncf-io\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-system-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-system-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-k8s-cni-cncf-io\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.370562 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-netns\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-netns\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcdb\" (UniqueName: \"kubernetes.io/projected/d27b7496-86c5-4818-89b4-8d50a41259bc-kube-api-access-pwcdb\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370593 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-cni-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.370649 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.370647 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:18.870626559 +0000 UTC m=+3.076582307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-kubelet\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-dbus\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-etc-kubernetes\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-socket-dir-parent\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-kubelet\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-etc-kubernetes\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-socket-dir-parent\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-dbus\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-conf-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-kubelet-config\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-bin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-conf-dir\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-hostroot\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-daemon-config\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-bin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-multus-certs\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-hostroot\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-cnibin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.370955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-kubelet-config\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-cni-binary-copy\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-run-multus-certs\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-cnibin\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-multus\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.371846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.371109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d27b7496-86c5-4818-89b4-8d50a41259bc-host-var-lib-cni-multus\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.372181 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.372161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-multus-daemon-config\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.372258 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.372239 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d27b7496-86c5-4818-89b4-8d50a41259bc-cni-binary-copy\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.387305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.387238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcdb\" (UniqueName: \"kubernetes.io/projected/d27b7496-86c5-4818-89b4-8d50a41259bc-kube-api-access-pwcdb\") pod \"multus-czckf\" (UID: \"d27b7496-86c5-4818-89b4-8d50a41259bc\") " pod="openshift-multus/multus-czckf" Sep 16 19:05:18.458870 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.458832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" Sep 16 19:05:18.466579 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.466547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9fwsh" Sep 16 19:05:18.475470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.475439 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj67g" Sep 16 19:05:18.480618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.480584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:18.488645 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.488624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" Sep 16 19:05:18.494670 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.494648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:18.502529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.502506 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bc4k" Sep 16 19:05:18.508416 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.508396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" Sep 16 19:05:18.514277 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.514260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czckf" Sep 16 19:05:18.775849 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.775754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:18.776020 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.775948 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:18.776020 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.776018 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:19.776002702 +0000 UTC m=+3.981958435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:18.876781 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.876709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:18.876945 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:18.876924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:18.876986 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.876947 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:18.876986 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.876982 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:18.877108 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.877021 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:18.877108 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.877030 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:18.877108 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.877091 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:19.877070789 +0000 UTC m=+4.083026527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:18.877254 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:18.877112 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:19.877101319 +0000 UTC m=+4.083057054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:19.095775 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.095748 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5a33ab_faac_469c_8a6d_f995291ddffc.slice/crio-b51f51811b71e2a48aaa7a08940a83d9f2b564a190119538c46bf21d82ab547e WatchSource:0}: Error finding container b51f51811b71e2a48aaa7a08940a83d9f2b564a190119538c46bf21d82ab547e: Status 404 returned error can't find the container with id b51f51811b71e2a48aaa7a08940a83d9f2b564a190119538c46bf21d82ab547e Sep 16 19:05:19.100209 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.100181 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da48f5d_46b1_4fe4_b6e8_3aa0640f74de.slice/crio-269adbf6253ed110e23deb330c85616c4eef2ec993a305e0a4c27500ad49917f WatchSource:0}: Error finding container 269adbf6253ed110e23deb330c85616c4eef2ec993a305e0a4c27500ad49917f: Status 404 returned error can't find the container with id 269adbf6253ed110e23deb330c85616c4eef2ec993a305e0a4c27500ad49917f Sep 16 19:05:19.101483 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.101449 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297542fb_6cc8_409f_a483_4749340921ef.slice/crio-f33cae72a94d59985365b6b35d2e4e324a6b82a77a93e0a17386084d64a494d1 WatchSource:0}: Error finding container f33cae72a94d59985365b6b35d2e4e324a6b82a77a93e0a17386084d64a494d1: Status 404 returned error can't find the container with id f33cae72a94d59985365b6b35d2e4e324a6b82a77a93e0a17386084d64a494d1 Sep 16 19:05:19.102843 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.102820 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71032cf_e13b_454c_9b19_9cd8e1ade547.slice/crio-70de12001bae1bed0dac8cba3f6d3fd00ac50abbe6333c069912cba8e699ef05 WatchSource:0}: Error finding container 70de12001bae1bed0dac8cba3f6d3fd00ac50abbe6333c069912cba8e699ef05: Status 404 returned error can't find the container with id 70de12001bae1bed0dac8cba3f6d3fd00ac50abbe6333c069912cba8e699ef05 Sep 16 19:05:19.103701 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.103675 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27b7496_86c5_4818_89b4_8d50a41259bc.slice/crio-6604c12a97c7a5f8db10d846d5cf14c8208aa337b080be87f165842e5994e527 WatchSource:0}: Error finding container 6604c12a97c7a5f8db10d846d5cf14c8208aa337b080be87f165842e5994e527: Status 404 returned error can't find the container with id 6604c12a97c7a5f8db10d846d5cf14c8208aa337b080be87f165842e5994e527 Sep 16 19:05:19.105802 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.105777 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f5c18c_6c8b_4cc8_b36f_67dba09488e7.slice/crio-6464e8a703ab51b6aeca4bb90b617f4c4edd2428e8275e3c959529cf9e6d7df1 WatchSource:0}: Error finding container 6464e8a703ab51b6aeca4bb90b617f4c4edd2428e8275e3c959529cf9e6d7df1: Status 404 returned error can't find the container with id 6464e8a703ab51b6aeca4bb90b617f4c4edd2428e8275e3c959529cf9e6d7df1 Sep 16 19:05:19.106519 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.106496 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb895f80_eded_4f74_a5f9_bfeeec6bc53e.slice/crio-7dedceb3bec874f9555cbd61e7200c5b0233f6e44c71cf00bc8d9fa86644b0e2 WatchSource:0}: Error finding container 7dedceb3bec874f9555cbd61e7200c5b0233f6e44c71cf00bc8d9fa86644b0e2: Status 404 returned error can't find the container with id 7dedceb3bec874f9555cbd61e7200c5b0233f6e44c71cf00bc8d9fa86644b0e2 Sep 16 19:05:19.108227 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.108198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd60af3_1dd1_4bb8_a22c_c821d2c69412.slice/crio-2b7e7af1bde144132f1f145cd51a256ef24f281ad4f843f7679910259324fc3c WatchSource:0}: Error finding container 2b7e7af1bde144132f1f145cd51a256ef24f281ad4f843f7679910259324fc3c: Status 404 returned error can't find the container with id 2b7e7af1bde144132f1f145cd51a256ef24f281ad4f843f7679910259324fc3c Sep 16 19:05:19.109538 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:19.109470 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb547de82_20e4_4a98_b580_70f04676b620.slice/crio-c1acb57f146b3aa691ad81438b5edab4f88c5390bf72b1f5b085b7d762476ebd WatchSource:0}: Error finding container c1acb57f146b3aa691ad81438b5edab4f88c5390bf72b1f5b085b7d762476ebd: Status 404 returned error can't find the container with id c1acb57f146b3aa691ad81438b5edab4f88c5390bf72b1f5b085b7d762476ebd Sep 16 19:05:19.203492 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.203425 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:00:18 +0000 UTC" deadline="2028-06-05 16:13:53.522950282 +0000 UTC" Sep 16 19:05:19.203492 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.203489 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15069h8m34.319465952s" Sep 16 19:05:19.255947 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.255911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fwsh" event={"ID":"b547de82-20e4-4a98-b580-70f04676b620","Type":"ContainerStarted","Data":"c1acb57f146b3aa691ad81438b5edab4f88c5390bf72b1f5b085b7d762476ebd"} Sep 16 19:05:19.257359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.257304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czckf" event={"ID":"d27b7496-86c5-4818-89b4-8d50a41259bc","Type":"ContainerStarted","Data":"6604c12a97c7a5f8db10d846d5cf14c8208aa337b080be87f165842e5994e527"} Sep 16 19:05:19.258413 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.258391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" event={"ID":"cf5a33ab-faac-469c-8a6d-f995291ddffc","Type":"ContainerStarted","Data":"b51f51811b71e2a48aaa7a08940a83d9f2b564a190119538c46bf21d82ab547e"} Sep 16 19:05:19.260213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.260191 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" event={"ID":"f1353260c7ace9243a51756424e4b1aa","Type":"ContainerStarted","Data":"7904246a30def1be55648f19d8d5f51ba24b3c40c3a77c1727b2b92bd37cf706"} Sep 16 19:05:19.261330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.261309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" event={"ID":"8bd60af3-1dd1-4bb8-a22c-c821d2c69412","Type":"ContainerStarted","Data":"2b7e7af1bde144132f1f145cd51a256ef24f281ad4f843f7679910259324fc3c"} Sep 16 19:05:19.262288 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.262270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bc4k" event={"ID":"cb895f80-eded-4f74-a5f9-bfeeec6bc53e","Type":"ContainerStarted","Data":"7dedceb3bec874f9555cbd61e7200c5b0233f6e44c71cf00bc8d9fa86644b0e2"} Sep 16 19:05:19.263277 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.263258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkrhj" event={"ID":"57f5c18c-6c8b-4cc8-b36f-67dba09488e7","Type":"ContainerStarted","Data":"6464e8a703ab51b6aeca4bb90b617f4c4edd2428e8275e3c959529cf9e6d7df1"} Sep 16 19:05:19.264250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.264231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"70de12001bae1bed0dac8cba3f6d3fd00ac50abbe6333c069912cba8e699ef05"} Sep 16 19:05:19.265386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.265356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj67g" event={"ID":"297542fb-6cc8-409f-a483-4749340921ef","Type":"ContainerStarted","Data":"f33cae72a94d59985365b6b35d2e4e324a6b82a77a93e0a17386084d64a494d1"} Sep 16 19:05:19.266358 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.266341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerStarted","Data":"269adbf6253ed110e23deb330c85616c4eef2ec993a305e0a4c27500ad49917f"} Sep 16 19:05:19.785129 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.785006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:19.785297 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.785223 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:19.785297 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.785291 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:21.785271085 +0000 UTC m=+5.991226833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:19.886481 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.886441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:19.886642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:19.886540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:19.886710 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.886675 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:19.886772 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.886738 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:21.886719485 +0000 UTC m=+6.092675231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:19.887152 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.887131 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:19.887214 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.887162 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:19.887214 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.887175 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:19.887350 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:19.887223 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:21.887207695 +0000 UTC m=+6.093163465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:20.203905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.203810 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:00:18 +0000 UTC" deadline="2028-05-26 20:45:32.040407358 +0000 UTC" Sep 16 19:05:20.203905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.203854 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14833h40m11.836557796s" Sep 16 19:05:20.249443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.249411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:20.249443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.249438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:20.249669 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.249432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:20.249669 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:20.249544 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:20.249669 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:20.249652 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:20.249814 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:20.249738 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:20.277365 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.277280 2565 generic.go:356] "Generic (PLEG): container finished" podID="0a59d530e718667cce688f2a3ad43aaa" containerID="e4ce0e9b1781b365735d1f16d8867d4291828c9d5e88d9e135ae85d93aa1e639" exitCode=0 Sep 16 19:05:20.277930 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.277899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerDied","Data":"e4ce0e9b1781b365735d1f16d8867d4291828c9d5e88d9e135ae85d93aa1e639"} Sep 16 19:05:20.326485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:20.324141 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" podStartSLOduration=3.324116134 podStartE2EDuration="3.324116134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:19.302675822 +0000 UTC m=+3.508631579" watchObservedRunningTime="2026-09-16 19:05:20.324116134 +0000 UTC m=+4.530071876" Sep 16 19:05:21.298569 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:21.298528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerStarted","Data":"449b408774e953c19078f82b68cd4cc1fd0814f150a0218cc15b097b0639d62f"} Sep 16 19:05:21.802642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:21.802043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:21.802642 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.802232 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:21.802642 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.802311 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.802275822 +0000 UTC m=+10.008231558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:21.902910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:21.903003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903146 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903207 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.903189631 +0000 UTC m=+10.109145380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903305 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903320 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903333 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:21.903725 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:21.903367 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:25.903356216 +0000 UTC m=+10.109311962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:22.247157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:22.247091 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:22.247700 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:22.247449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:22.247700 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:22.247530 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:22.247700 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:22.247452 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:22.247700 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:22.247594 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:22.247700 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:22.247670 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:24.247408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:24.247374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:24.247858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:24.247374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:24.247858 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:24.247517 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:24.247858 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:24.247550 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:24.247858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:24.247607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:24.247858 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:24.247684 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:25.837146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:25.837112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:25.837617 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.837306 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:25.837617 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.837380 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.837365569 +0000 UTC m=+18.043321303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:25.938275 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:25.938237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:25.938475 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:25.938361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:25.938475 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938422 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:25.938475 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938447 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:25.938475 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938457 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:25.938734 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938479 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:25.938734 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938516 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.938498826 +0000 UTC m=+18.144454564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:25.938734 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:25.938532 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.938524525 +0000 UTC m=+18.144480256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:26.249066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:26.248986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:26.249212 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:26.249116 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:26.249495 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:26.249474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:26.249616 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:26.249573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:26.249683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:26.249623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:26.249738 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:26.249688 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:28.247583 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:28.247541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:28.247583 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:28.247575 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:28.247583 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:28.247541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:28.248162 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:28.247693 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:28.248162 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:28.247779 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:28.248162 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:28.247808 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:30.247058 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:30.247102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:30.247067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:30.247205 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:30.247293 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:30.247599 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:30.247366 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:32.247382 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:32.247348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:32.247818 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:32.247349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:32.247818 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:32.247474 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:32.247818 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:32.247357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:32.247818 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:32.247594 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:32.247818 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:32.247642 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:33.898191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:33.898090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:33.898675 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.898218 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:33.898675 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.898298 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:05:49.898265072 +0000 UTC m=+34.104220805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:33.999059 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:33.999014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:33.999059 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:33.999064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999168 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999178 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999198 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999208 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999249 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:49.999227747 +0000 UTC m=+34.205183481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:33.999314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:33.999264 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:49.999258404 +0000 UTC m=+34.205214135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:34.247030 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:34.246948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:34.247198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:34.246952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:34.247198 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:34.247065 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:34.247198 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:34.247170 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:34.247198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:34.246952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:34.247431 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:34.247329 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:36.248451 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:36.248411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:36.248919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:36.248490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:36.248919 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:36.248507 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:36.248919 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:36.248599 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:36.248919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:36.248627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:36.248919 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:36.248704 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:38.248162 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.247693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:38.248830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.247728 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:38.248830 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:38.248180 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:38.248830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.247771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:38.248830 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:38.248483 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:38.248830 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:38.248519 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:38.335469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.335407 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj67g" event={"ID":"297542fb-6cc8-409f-a483-4749340921ef","Type":"ContainerStarted","Data":"aa28464a741606f3d009f7ca99ff44f3c080f4d7cd224728f4f2aaf7fd5e3f5c"} Sep 16 19:05:38.337144 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.337117 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="7d3f82437c4da29a3d09a29bf9c6b95f4a07d345fbc010bce6ed16edd7430c78" exitCode=0 Sep 16 19:05:38.337268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.337195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"7d3f82437c4da29a3d09a29bf9c6b95f4a07d345fbc010bce6ed16edd7430c78"} Sep 16 19:05:38.339246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.339200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9fwsh" event={"ID":"b547de82-20e4-4a98-b580-70f04676b620","Type":"ContainerStarted","Data":"db0fa8d30e9f37ce1569ed1a77abed19b57d8370953338a36ae3129c135a4a6c"} Sep 16 19:05:38.341088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.341043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czckf" event={"ID":"d27b7496-86c5-4818-89b4-8d50a41259bc","Type":"ContainerStarted","Data":"70bf39abeaadb5b8173d7bb203189ad78877e81d5c6a58608c2da847bc3fa8f3"} Sep 16 19:05:38.342726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.342706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" event={"ID":"cf5a33ab-faac-469c-8a6d-f995291ddffc","Type":"ContainerStarted","Data":"0fc22268ac1c7c96abcafd7ab450c8011938225ad77f82a3754d18928cb63459"} Sep 16 19:05:38.344448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.344428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" event={"ID":"8bd60af3-1dd1-4bb8-a22c-c821d2c69412","Type":"ContainerStarted","Data":"2265f56aaf3072354205ec454dfa63cc9f582a3d6e92627b26b72092ba6d3c00"} Sep 16 19:05:38.346079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.346057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkrhj" event={"ID":"57f5c18c-6c8b-4cc8-b36f-67dba09488e7","Type":"ContainerStarted","Data":"5a2143b725f57dbb05fa122fb6b4b39aaa04e13d21458a9745d2244fb42e2657"} Sep 16 19:05:38.349170 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"16926e636800cb9b7d909e80ac023ed3739a96abc2b0d5c473e1bf48ce44d6b0"} Sep 16 19:05:38.349233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"02d40c9862339d0f23d8c1146219fed0864710812be20f6f866e5714346f0da6"} Sep 16 19:05:38.349305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349232 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"0b1da673ccfdb9d57f8468097f20960c7fe160c9e61c97f4589bcb3af5299618"} Sep 16 19:05:38.349305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"cd75d231a216f7ab330be48229b84d66e3d1d53bf1ebec8b53e760da78c096b2"} Sep 16 19:05:38.349390 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"1adeedebe92d74396f85268f977921c2fba205a687cb4852ab05e1c955a1a1d9"} Sep 16 19:05:38.349390 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.349339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"892bbe800b552129225d1b20f3eda2fa1c869ed2d3384fd34f4a7ef14249224a"} Sep 16 19:05:38.356087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.356044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mj67g" podStartSLOduration=3.962626196 podStartE2EDuration="22.35602735s" totalImagesPullingTime="18.393401154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:38.355775084 +0000 UTC m=+22.561730840" watchObservedRunningTime="2026-09-16 19:05:38.35602735 +0000 UTC m=+22.561983107" Sep 16 19:05:38.356171 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.356141 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" podStartSLOduration=21.356132379 podStartE2EDuration="21.356132379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:21.325021214 +0000 UTC m=+5.530976970" watchObservedRunningTime="2026-09-16 19:05:38.356132379 +0000 UTC m=+22.562088133" Sep 16 19:05:38.376156 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.376111 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nb9ft" podStartSLOduration=3.989979645 podStartE2EDuration="22.37609421s" totalImagesPullingTime="18.386114565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:38.375862289 +0000 UTC m=+22.581818042" watchObservedRunningTime="2026-09-16 19:05:38.37609421 +0000 UTC m=+22.582050014" Sep 16 19:05:38.394440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.394400 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9fwsh" podStartSLOduration=4.014747807 podStartE2EDuration="22.394385293s" totalImagesPullingTime="18.379637486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:38.394345092 +0000 UTC m=+22.600300842" watchObservedRunningTime="2026-09-16 19:05:38.394385293 +0000 UTC m=+22.600341044" Sep 16 19:05:38.445108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.445053 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkrhj" podStartSLOduration=8.046584917 podStartE2EDuration="22.445032902s" totalImagesPullingTime="14.398447985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:38.444868921 +0000 UTC m=+22.650824675" watchObservedRunningTime="2026-09-16 19:05:38.445032902 +0000 UTC m=+22.650988655" Sep 16 19:05:38.476083 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.476035 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-czckf" podStartSLOduration=4.024277599 podStartE2EDuration="22.47602207s" totalImagesPullingTime="18.451744471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:38.475946827 +0000 UTC m=+22.681902639" watchObservedRunningTime="2026-09-16 19:05:38.47602207 +0000 UTC m=+22.681977820" Sep 16 19:05:38.909946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:38.909915 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 19:05:39.222492 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.222337 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:05:38.909940464Z","UUID":"0304a785-b771-490f-9e51-fcffea8564a4","Handler":null,"Name":"","Endpoint":""} Sep 16 19:05:39.224079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.224060 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 19:05:39.224194 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.224091 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 19:05:39.352941 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.352871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bc4k" event={"ID":"cb895f80-eded-4f74-a5f9-bfeeec6bc53e","Type":"ContainerStarted","Data":"4f0f93108b155ff2a74462f78bff5aff51331a433a1b046ad17fa216dda16c0c"} Sep 16 19:05:39.355587 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.355548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" event={"ID":"cf5a33ab-faac-469c-8a6d-f995291ddffc","Type":"ContainerStarted","Data":"6a826408f2820b3839aa61bb4a8c237568be79e4014b12acd60ed7e0f9fb5a07"} Sep 16 19:05:39.378210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:39.378158 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9bc4k" podStartSLOduration=4.993540357 podStartE2EDuration="23.378142897s" totalImagesPullingTime="18.38460254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:39.377453318 +0000 UTC m=+23.583409073" watchObservedRunningTime="2026-09-16 19:05:39.378142897 +0000 UTC m=+23.584098650" Sep 16 19:05:40.084494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.084462 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:40.085114 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.085095 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:40.246985 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.246957 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:40.247117 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:40.247090 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:40.247525 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.247492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:40.247621 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:40.247576 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:40.247682 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.247630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:40.247737 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:40.247677 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:40.360356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.360320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" event={"ID":"cf5a33ab-faac-469c-8a6d-f995291ddffc","Type":"ContainerStarted","Data":"ea2cc2b564c2098ecd61d2ab5c6c0a90b0c78e86cf87d34a0b7a7b1c386b896e"} Sep 16 19:05:40.364936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.364907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"f2ab9a28bbbc5cf96f69017b5d5f88cf3188a508bb266b2bf95de7237c51ed1e"} Sep 16 19:05:40.365357 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.365335 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:40.365977 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.365960 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkrhj" Sep 16 19:05:40.438813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:40.438752 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdzgp" podStartSLOduration=3.941552907 podStartE2EDuration="24.438726817s" totalImagesPullingTime="20.49717391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:40.395363421 +0000 UTC m=+24.601319173" watchObservedRunningTime="2026-09-16 19:05:40.438726817 +0000 UTC m=+24.644682571" Sep 16 19:05:42.247116 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:42.247083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:42.247596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:42.247083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:42.247596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:42.247226 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:42.247596 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:42.247220 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:42.247596 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:42.247388 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:42.247596 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:42.247443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:43.375357 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.375136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" event={"ID":"d71032cf-e13b-454c-9b19-9cd8e1ade547","Type":"ContainerStarted","Data":"17cf168cf7072bf96eb361d3dd05d97beac770b199a8bde3f1dc27207f14dc04"} Sep 16 19:05:43.376229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.375436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:43.376229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.375494 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:43.376229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.375541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:43.377773 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.377744 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="902d4d59597692b9456ae0996524c8e748f06e999ee4a0147b115ddbce177589" exitCode=0 Sep 16 19:05:43.377913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.377796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"902d4d59597692b9456ae0996524c8e748f06e999ee4a0147b115ddbce177589"} Sep 16 19:05:43.391482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.391451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:43.391618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.391601 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:05:43.429695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:43.429651 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" podStartSLOduration=8.986691094 podStartE2EDuration="27.429636146s" totalImagesPullingTime="18.442945052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:43.429100433 +0000 UTC m=+27.635056187" watchObservedRunningTime="2026-09-16 19:05:43.429636146 +0000 UTC m=+27.635591898" Sep 16 19:05:44.247729 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.247697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:44.247729 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.247714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:44.248029 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.247815 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:44.248029 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.247882 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:44.248029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.247940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:44.248029 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.248022 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:44.616209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.615928 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpgbn"] Sep 16 19:05:44.616712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.616222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:44.616712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.616331 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:44.616712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.616649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kg6ff"] Sep 16 19:05:44.616895 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.616736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:44.616895 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.616820 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:44.632193 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.632165 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vnwn8"] Sep 16 19:05:44.632349 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:44.632258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:44.632396 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:44.632361 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:45.384663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:45.384626 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="3aee6804937d679f8a6681296b83ea3a5ffbe5490824d48786fd96b5be1fd664" exitCode=0 Sep 16 19:05:45.384820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:45.384718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"3aee6804937d679f8a6681296b83ea3a5ffbe5490824d48786fd96b5be1fd664"} Sep 16 19:05:46.247422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:46.247388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:46.247836 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:46.247462 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:46.247836 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:46.247514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:46.247836 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:46.247597 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:46.247836 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:46.247642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:46.247836 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:46.247773 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:47.391854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:47.391623 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="ae5cdc0f569c9d18f6c88ff9aeda986d6fc14b11580aea4e55cc5c461c0ce07a" exitCode=0 Sep 16 19:05:47.391854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:47.391711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"ae5cdc0f569c9d18f6c88ff9aeda986d6fc14b11580aea4e55cc5c461c0ce07a"} Sep 16 19:05:48.247666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:48.247628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:48.247857 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:48.247754 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:48.247857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:48.247628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:48.247857 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:48.247834 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:48.247857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:48.247628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:48.248048 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:48.247912 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:49.913531 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:49.913499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:49.913942 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:49.913632 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:49.913942 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:49.913699 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:06:21.913678288 +0000 UTC m=+66.119634032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 19:05:50.013811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.013779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.013837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.013889 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.013935 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.013955 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.013962 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret podName:13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:22.013945186 +0000 UTC m=+66.219900937 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret") pod "global-pull-secret-syncer-cpgbn" (UID: "13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830") : object "kube-system"/"original-pull-secret" not registered Sep 16 19:05:50.013974 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.013968 2565 projected.go:196] Error preparing data for projected volume kube-api-access-j7wjp for pod openshift-network-diagnostics/network-check-target-kg6ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:50.014205 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.014004 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp podName:c870367a-3802-412b-a96c-1ee6eb7a781f nodeName:}" failed. No retries permitted until 2026-09-16 19:06:22.01399414 +0000 UTC m=+66.219949885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7wjp" (UniqueName: "kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp") pod "network-check-target-kg6ff" (UID: "c870367a-3802-412b-a96c-1ee6eb7a781f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 19:05:50.247808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.247770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:50.248003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.247770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:50.248003 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.247911 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpgbn" podUID="13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830" Sep 16 19:05:50.248003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.247776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:50.248003 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.247985 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg6ff" podUID="c870367a-3802-412b-a96c-1ee6eb7a781f" Sep 16 19:05:50.248190 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.248059 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vnwn8" podUID="b4ea1abe-95f7-4dbc-b130-c3b9b471452a" Sep 16 19:05:50.520418 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.520328 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeReady" Sep 16 19:05:50.520563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.520466 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 19:05:50.614349 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.614315 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-969r4"] Sep 16 19:05:50.616881 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.616855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.617793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.617769 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wvl7c"] Sep 16 19:05:50.619801 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.619781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:50.625418 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625387 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 16 19:05:50.625418 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625387 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" Sep 16 19:05:50.625536 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625387 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 16 19:05:50.625536 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625442 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-d25ds\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d25ds\"" type="*v1.Secret" Sep 16 19:05:50.625536 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625447 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" Sep 16 19:05:50.625858 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.625829 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-default-969r4\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="8452df11-18f3-46c0-9353-920103de5013" pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.626115 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.626088 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-6pvzq\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\"" type="*v1.Secret" Sep 16 19:05:50.626354 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.626331 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" Sep 16 19:05:50.632143 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:50.632116 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-default-969r4\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="8452df11-18f3-46c0-9353-920103de5013" pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.646058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.646035 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-969r4"] Sep 16 19:05:50.689768 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.689733 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wvl7c"] Sep 16 19:05:50.719195 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.719337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-cert\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:50.719337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8452df11-18f3-46c0-9353-920103de5013-tmp-dir\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.719445 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jptkv\" (UniqueName: \"kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:50.719445 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdcc\" (UniqueName: \"kubernetes.io/projected/8452df11-18f3-46c0-9353-920103de5013-kube-api-access-mkdcc\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.719524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.719451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8452df11-18f3-46c0-9353-920103de5013-config-volume\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.820626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8452df11-18f3-46c0-9353-920103de5013-tmp-dir\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.820626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jptkv\" (UniqueName: \"kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:50.820830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdcc\" (UniqueName: \"kubernetes.io/projected/8452df11-18f3-46c0-9353-920103de5013-kube-api-access-mkdcc\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.820830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8452df11-18f3-46c0-9353-920103de5013-config-volume\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.820830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.820830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-cert\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:50.821003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.820938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8452df11-18f3-46c0-9353-920103de5013-tmp-dir\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:50.835902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:50.835871 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdcc\" (UniqueName: \"kubernetes.io/projected/8452df11-18f3-46c0-9353-920103de5013-kube-api-access-mkdcc\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:51.404803 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.404760 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerStarted","Data":"f3f5a594f3e36ba42464d82aea3789250d1ef15cddcaf4f60ab0f304871f8d7a"} Sep 16 19:05:51.433863 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.433843 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 19:05:51.441800 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:51.441776 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:51.441900 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:51.441840 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:51.941823062 +0000 UTC m=+36.147778792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:05:51.650971 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.650880 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 19:05:51.653681 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.653651 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-cert\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:51.707479 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.707448 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 19:05:51.802402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.802372 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 19:05:51.812301 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.812248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8452df11-18f3-46c0-9353-920103de5013-config-volume\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:51.835865 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:51.835827 2565 projected.go:291] Couldn't get configMap openshift-ingress-canary/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 16 19:05:51.835981 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:51.835878 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jptkv for pod openshift-ingress-canary/ingress-canary-wvl7c: failed to sync configmap cache: timed out waiting for the condition Sep 16 19:05:51.835981 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:51.835944 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv podName:18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b nodeName:}" failed. No retries permitted until 2026-09-16 19:05:52.335924661 +0000 UTC m=+36.541880393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jptkv" (UniqueName: "kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv") pod "ingress-canary-wvl7c" (UID: "18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b") : failed to sync configmap cache: timed out waiting for the condition Sep 16 19:05:51.855997 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.855967 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d25ds\"" Sep 16 19:05:51.903686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:51.903610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 19:05:52.028432 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.028394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:52.028561 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:52.028508 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:52.028561 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:52.028560 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:53.028547506 +0000 UTC m=+37.234503237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:05:52.228830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.228804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\"" Sep 16 19:05:52.247036 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.247006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:05:52.247139 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.247006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:05:52.247204 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.247013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:05:52.252867 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.252828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\"" Sep 16 19:05:52.252867 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.252855 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 19:05:52.253015 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.252887 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 19:05:52.253015 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.252834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 19:05:52.253730 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.253708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 19:05:52.256901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.256883 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\"" Sep 16 19:05:52.410402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.410367 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="f3f5a594f3e36ba42464d82aea3789250d1ef15cddcaf4f60ab0f304871f8d7a" exitCode=0 Sep 16 19:05:52.410735 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.410420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"f3f5a594f3e36ba42464d82aea3789250d1ef15cddcaf4f60ab0f304871f8d7a"} Sep 16 19:05:52.429843 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.429819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jptkv\" (UniqueName: \"kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:52.433988 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.433939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jptkv\" (UniqueName: \"kubernetes.io/projected/18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b-kube-api-access-jptkv\") pod \"ingress-canary-wvl7c\" (UID: \"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b\") " pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:52.434686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.434663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wvl7c" Sep 16 19:05:52.740005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:52.739731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wvl7c"] Sep 16 19:05:52.743509 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:05:52.743483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bd5854_61a8_4b7e_a7c1_e847f7dfaa3b.slice/crio-68ebd52fe31595bad1ef9bbd01ed14b7ae13b3c12cc7567fb1b44dc1e15dfa2b WatchSource:0}: Error finding container 68ebd52fe31595bad1ef9bbd01ed14b7ae13b3c12cc7567fb1b44dc1e15dfa2b: Status 404 returned error can't find the container with id 68ebd52fe31595bad1ef9bbd01ed14b7ae13b3c12cc7567fb1b44dc1e15dfa2b Sep 16 19:05:53.034210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:53.034119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:53.034370 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:53.034274 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:53.034370 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:53.034356 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:55.034338651 +0000 UTC m=+39.240294386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:05:53.414677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:53.414581 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wvl7c" event={"ID":"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b","Type":"ContainerStarted","Data":"68ebd52fe31595bad1ef9bbd01ed14b7ae13b3c12cc7567fb1b44dc1e15dfa2b"} Sep 16 19:05:53.418220 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:53.418183 2565 generic.go:356] "Generic (PLEG): container finished" podID="5da48f5d-46b1-4fe4-b6e8-3aa0640f74de" containerID="fa938f71f10a1e7b65f963ff594e0cb362c273be608ac2deb1ffee41f04fe7db" exitCode=0 Sep 16 19:05:53.418348 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:53.418223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerDied","Data":"fa938f71f10a1e7b65f963ff594e0cb362c273be608ac2deb1ffee41f04fe7db"} Sep 16 19:05:54.426343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:54.426250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" event={"ID":"5da48f5d-46b1-4fe4-b6e8-3aa0640f74de","Type":"ContainerStarted","Data":"ac1f13dc7891a8df918c1306dcbf0d4e1fe5cb44da0e6a2a57775847d85b6536"} Sep 16 19:05:54.454727 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:54.454414 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fp9xj" podStartSLOduration=9.905776472 podStartE2EDuration="38.454395435s" totalImagesPullingTime="28.548618963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:54.452712366 +0000 UTC m=+38.658668146" watchObservedRunningTime="2026-09-16 19:05:54.454395435 +0000 UTC m=+38.660351188" Sep 16 19:05:55.048826 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:55.048782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:55.049043 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:55.048913 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:55.049043 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:55.048964 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:59.048951289 +0000 UTC m=+43.254907020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:05:55.433614 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:55.433413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wvl7c" event={"ID":"18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b","Type":"ContainerStarted","Data":"1b4f255bc2069ff19b36afdba10c22837d2c2e760b9e91955fe7bec0017119e0"} Sep 16 19:05:55.454948 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:55.454885 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wvl7c" podStartSLOduration=2.996250481 podStartE2EDuration="5.454870581s" totalImagesPullingTime="2.4586201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:55.454669427 +0000 UTC m=+39.660625181" watchObservedRunningTime="2026-09-16 19:05:55.454870581 +0000 UTC m=+39.660826334" Sep 16 19:05:59.077415 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:05:59.077371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:05:59.077803 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:59.077494 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:05:59.077803 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:05:59.077547 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:07.077533777 +0000 UTC m=+51.283489511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:06:07.129180 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:07.129148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:06:07.129657 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:07.129267 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:06:07.129657 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:07.129329 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:23.12931705 +0000 UTC m=+67.335272781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:06:15.404587 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:15.404558 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hkdmc" Sep 16 19:06:21.934593 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:21.934556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:06:21.937156 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:21.937134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 19:06:21.945476 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:21.945456 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 19:06:21.945580 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:21.945514 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs podName:b4ea1abe-95f7-4dbc-b130-c3b9b471452a nodeName:}" failed. No retries permitted until 2026-09-16 19:07:25.945500444 +0000 UTC m=+130.151456175 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs") pod "network-metrics-daemon-vnwn8" (UID: "b4ea1abe-95f7-4dbc-b130-c3b9b471452a") : secret "metrics-daemon-secret" not found Sep 16 19:06:22.035447 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.035412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:06:22.035627 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.035460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:06:22.039092 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.039075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 19:06:22.039174 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.039082 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 19:06:22.048968 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.048943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 19:06:22.050085 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.050064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830-original-pull-secret\") pod \"global-pull-secret-syncer-cpgbn\" (UID: \"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830\") " pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:06:22.059295 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.059261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wjp\" (UniqueName: \"kubernetes.io/projected/c870367a-3802-412b-a96c-1ee6eb7a781f-kube-api-access-j7wjp\") pod \"network-check-target-kg6ff\" (UID: \"c870367a-3802-412b-a96c-1ee6eb7a781f\") " pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:06:22.258222 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.258192 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpgbn" Sep 16 19:06:22.271601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.271575 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45r44\"" Sep 16 19:06:22.278742 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.278727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:06:22.569975 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.569896 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpgbn"] Sep 16 19:06:22.572697 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:06:22.572670 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a5cf8d_5203_4fd9_b4cc_03d2d8a0c830.slice/crio-061617bd53bca644fa1bc3253da7dfec4fa3a5f630732ad542222abb649964af WatchSource:0}: Error finding container 061617bd53bca644fa1bc3253da7dfec4fa3a5f630732ad542222abb649964af: Status 404 returned error can't find the container with id 061617bd53bca644fa1bc3253da7dfec4fa3a5f630732ad542222abb649964af Sep 16 19:06:22.586124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:22.586079 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kg6ff"] Sep 16 19:06:22.591500 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:06:22.591470 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc870367a_3802_412b_a96c_1ee6eb7a781f.slice/crio-1371183680e8019651eb0f79f8cc98af1d2164596c5a4e56bb1237af441d3715 WatchSource:0}: Error finding container 1371183680e8019651eb0f79f8cc98af1d2164596c5a4e56bb1237af441d3715: Status 404 returned error can't find the container with id 1371183680e8019651eb0f79f8cc98af1d2164596c5a4e56bb1237af441d3715 Sep 16 19:06:23.142003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:23.141965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:06:23.142454 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:23.142117 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:06:23.142454 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:23.142242 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:55.142218993 +0000 UTC m=+99.348174725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:06:23.501294 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:23.501235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpgbn" event={"ID":"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830","Type":"ContainerStarted","Data":"061617bd53bca644fa1bc3253da7dfec4fa3a5f630732ad542222abb649964af"} Sep 16 19:06:23.503098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:23.503059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kg6ff" event={"ID":"c870367a-3802-412b-a96c-1ee6eb7a781f","Type":"ContainerStarted","Data":"1371183680e8019651eb0f79f8cc98af1d2164596c5a4e56bb1237af441d3715"} Sep 16 19:06:27.518014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:27.517980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kg6ff" event={"ID":"c870367a-3802-412b-a96c-1ee6eb7a781f","Type":"ContainerStarted","Data":"917909a1b2e472d510a383c5e2de6504a9bd1bcdaafb7e7f1579fb49f06ed2c6"} Sep 16 19:06:27.518493 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:27.518115 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:06:27.519524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:27.519493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpgbn" event={"ID":"13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830","Type":"ContainerStarted","Data":"6953e78b77271680a094da65b95b4e5a666f4cbd94a1e776d7b249d4c6ccf178"} Sep 16 19:06:27.536967 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:27.536930 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kg6ff" podStartSLOduration=66.983196516 podStartE2EDuration="1m11.536917528s" totalImagesPullingTime="4.553721012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:27.536546536 +0000 UTC m=+71.742502288" watchObservedRunningTime="2026-09-16 19:06:27.536917528 +0000 UTC m=+71.742873311" Sep 16 19:06:27.553402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:27.553367 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cpgbn" podStartSLOduration=66.977841282 podStartE2EDuration="1m11.553357262s" totalImagesPullingTime="4.57551598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:27.552620279 +0000 UTC m=+71.758576032" watchObservedRunningTime="2026-09-16 19:06:27.553357262 +0000 UTC m=+71.759313014" Sep 16 19:06:42.107616 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.107492 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74"] Sep 16 19:06:42.111552 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.111536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" Sep 16 19:06:42.119518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.119498 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 19:06:42.119625 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.119497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q75nx\"" Sep 16 19:06:42.119925 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.119910 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:06:42.134723 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.134700 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cf589d94b-rxcqc"] Sep 16 19:06:42.137448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.137434 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.150578 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.150544 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-metrics-certs-default\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" Sep 16 19:06:42.150578 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.150559 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-stats-default\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" Sep 16 19:06:42.150752 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.150580 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-dockercfg-4bj6c\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4bj6c\"" type="*v1.Secret" Sep 16 19:06:42.150752 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.150739 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 19:06:42.150894 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.150564 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" Sep 16 19:06:42.151007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.150956 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 19:06:42.151063 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.151012 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 19:06:42.151446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.151430 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-5lz2w"] Sep 16 19:06:42.154373 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.154356 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74"] Sep 16 19:06:42.154490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.154455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.156461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156442 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 19:06:42.156652 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-tmp\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-snapshots\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-service-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.156724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k9qlj\"" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzh2\" (UniqueName: \"kubernetes.io/projected/70722144-5165-41ed-9e93-0c3fd3354607-kube-api-access-mzzh2\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-trusted-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aed24638-847e-4542-82ad-ac5eee3df193-serving-cert\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchnb\" (UniqueName: \"kubernetes.io/projected/b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc-kube-api-access-mchnb\") pod \"volume-data-source-validator-86695b7cc7-2rb74\" (UID: \"b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkdz\" (UniqueName: \"kubernetes.io/projected/aed24638-847e-4542-82ad-ac5eee3df193-kube-api-access-slkdz\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.157041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.156882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-default-certificate\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.157230 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.157145 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 19:06:42.157702 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.157686 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 19:06:42.162699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.162682 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 19:06:42.174031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.174005 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 19:06:42.183802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.183781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cf589d94b-rxcqc"] Sep 16 19:06:42.207537 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.207506 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-5lz2w"] Sep 16 19:06:42.240722 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.240691 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:06:42.243458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.243442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.246507 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.246483 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 19:06:42.246507 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.246503 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 19:06:42.246667 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.246495 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pps2s\"" Sep 16 19:06:42.247199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.247183 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 19:06:42.258868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.258846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.259012 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.258883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.259012 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.258900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxqc\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.259012 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.258972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.259176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-tmp\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.259176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.259176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.259176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-snapshots\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-service-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzh2\" (UniqueName: \"kubernetes.io/projected/70722144-5165-41ed-9e93-0c3fd3354607-kube-api-access-mzzh2\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-trusted-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aed24638-847e-4542-82ad-ac5eee3df193-serving-cert\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchnb\" (UniqueName: \"kubernetes.io/projected/b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc-kube-api-access-mchnb\") pod \"volume-data-source-validator-86695b7cc7-2rb74\" (UID: \"b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" Sep 16 19:06:42.259443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkdz\" (UniqueName: \"kubernetes.io/projected/aed24638-847e-4542-82ad-ac5eee3df193-kube-api-access-slkdz\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-tmp\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-default-certificate\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aed24638-847e-4542-82ad-ac5eee3df193-snapshots\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.260087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.259857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-service-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.260692 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.260231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed24638-847e-4542-82ad-ac5eee3df193-trusted-ca-bundle\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.260692 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.260583 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 19:06:42.262059 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.262034 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aed24638-847e-4542-82ad-ac5eee3df193-serving-cert\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.262156 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.262137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-default-certificate\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.268936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.268915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:06:42.271323 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.271304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzh2\" (UniqueName: \"kubernetes.io/projected/70722144-5165-41ed-9e93-0c3fd3354607-kube-api-access-mzzh2\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:42.279470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.279445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchnb\" (UniqueName: \"kubernetes.io/projected/b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc-kube-api-access-mchnb\") pod \"volume-data-source-validator-86695b7cc7-2rb74\" (UID: \"b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" Sep 16 19:06:42.282092 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.282070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkdz\" (UniqueName: \"kubernetes.io/projected/aed24638-847e-4542-82ad-ac5eee3df193-kube-api-access-slkdz\") pod \"insights-operator-5844b856db-5lz2w\" (UID: \"aed24638-847e-4542-82ad-ac5eee3df193\") " pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.360053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.359973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.360099 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.360115 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxqc\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.360164 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:06:42.860148339 +0000 UTC m=+87.066104073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360353 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.360877 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.360857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.361371 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.361331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.362570 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.362550 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.362712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.362694 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.388073 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.388031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.388633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.388615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxqc\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.419961 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.419936 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" Sep 16 19:06:42.463187 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.463144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-5lz2w" Sep 16 19:06:42.711621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.711542 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74"] Sep 16 19:06:42.721332 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:06:42.721306 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e5ca5_b5a3_4f2d_811f_3b96aff46ddc.slice/crio-f40b61384a38c87cf122cbe237b6866b3ad42664db007204e7bcf07f50210129 WatchSource:0}: Error finding container f40b61384a38c87cf122cbe237b6866b3ad42664db007204e7bcf07f50210129: Status 404 returned error can't find the container with id f40b61384a38c87cf122cbe237b6866b3ad42664db007204e7bcf07f50210129 Sep 16 19:06:42.740462 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.740436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-5lz2w"] Sep 16 19:06:42.743164 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:06:42.743140 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed24638_847e_4542_82ad_ac5eee3df193.slice/crio-91f917bf2c5cd9156ddf95c03980f545054466811011613c76f25f0a76039a46 WatchSource:0}: Error finding container 91f917bf2c5cd9156ddf95c03980f545054466811011613c76f25f0a76039a46: Status 404 returned error can't find the container with id 91f917bf2c5cd9156ddf95c03980f545054466811011613c76f25f0a76039a46 Sep 16 19:06:42.864160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:42.864127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:42.864349 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.864266 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:42.864349 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.864304 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:42.864423 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:42.864355 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.864341091 +0000 UTC m=+88.070296822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:43.259356 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259319 2565 secret.go:189] Couldn't get secret openshift-ingress/router-stats-default: failed to sync secret cache: timed out waiting for the condition Sep 16 19:06:43.259356 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259340 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: failed to sync secret cache: timed out waiting for the condition Sep 16 19:06:43.259825 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259411 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.759389398 +0000 UTC m=+87.965345146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "stats-auth" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : failed to sync secret cache: timed out waiting for the condition Sep 16 19:06:43.259825 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259416 2565 configmap.go:193] Couldn't get configMap openshift-ingress/service-ca-bundle: failed to sync configmap cache: timed out waiting for the condition Sep 16 19:06:43.259825 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259431 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.759421729 +0000 UTC m=+87.965377462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : failed to sync secret cache: timed out waiting for the condition Sep 16 19:06:43.259825 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.259465 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:43.759452397 +0000 UTC m=+87.965408127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : failed to sync configmap cache: timed out waiting for the condition Sep 16 19:06:43.447239 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.447210 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 19:06:43.542522 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.542434 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 19:06:43.558406 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.558364 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-5lz2w" event={"ID":"aed24638-847e-4542-82ad-ac5eee3df193","Type":"ContainerStarted","Data":"91f917bf2c5cd9156ddf95c03980f545054466811011613c76f25f0a76039a46"} Sep 16 19:06:43.560264 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.560230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" event={"ID":"b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc","Type":"ContainerStarted","Data":"f40b61384a38c87cf122cbe237b6866b3ad42664db007204e7bcf07f50210129"} Sep 16 19:06:43.569814 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.569780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 19:06:43.609912 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.609879 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4bj6c\"" Sep 16 19:06:43.771921 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.771893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:43.772022 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.771945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:43.772022 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.771982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:43.772204 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.772190 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:44.772172916 +0000 UTC m=+88.978128650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : configmap references non-existent config key: service-ca.crt Sep 16 19:06:43.772486 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.772470 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 19:06:43.772556 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.772514 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:44.772503723 +0000 UTC m=+88.978459454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : secret "router-metrics-certs-default" not found Sep 16 19:06:43.775369 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.775345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-stats-auth\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:43.873182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:43.873147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:43.873347 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.873303 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:43.873347 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.873316 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:43.873444 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:43.873363 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:06:45.873350596 +0000 UTC m=+90.079306327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:44.564249 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:44.564215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" event={"ID":"b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc","Type":"ContainerStarted","Data":"b939f63a4c0560f37a2b25992fdf7219bb77d79bf1b8f79c11379dd7ad3876e4"} Sep 16 19:06:44.585698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:44.585648 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2rb74" podStartSLOduration=1.5539951730000001 podStartE2EDuration="2.58562991s" totalImagesPullingTime="1.031634737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:44.585620057 +0000 UTC m=+88.791575810" watchObservedRunningTime="2026-09-16 19:06:44.58562991 +0000 UTC m=+88.791585664" Sep 16 19:06:44.779667 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:44.779627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:44.779866 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:44.779708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:44.779866 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:44.779796 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 19:06:44.779866 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:44.779820 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.779799127 +0000 UTC m=+90.985754861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : configmap references non-existent config key: service-ca.crt Sep 16 19:06:44.779866 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:44.779850 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.779840662 +0000 UTC m=+90.985796395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : secret "router-metrics-certs-default" not found Sep 16 19:06:45.571334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.571275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-5lz2w" event={"ID":"aed24638-847e-4542-82ad-ac5eee3df193","Type":"ContainerStarted","Data":"2f17ac9671a365a1a0fcd1d796b166846a0ba46189fb6cb7d7bb2c25ea9549e9"} Sep 16 19:06:45.593509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.593457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-5lz2w" podStartSLOduration=1.751350254 podStartE2EDuration="3.593441229s" totalImagesPullingTime="1.842090975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:45.592928105 +0000 UTC m=+89.798883857" watchObservedRunningTime="2026-09-16 19:06:45.593441229 +0000 UTC m=+89.799396982" Sep 16 19:06:45.818040 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.818010 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mwzlf"] Sep 16 19:06:45.821233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.821213 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.824396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.824326 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 19:06:45.824396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.824380 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 19:06:45.824396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.824388 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27w29\"" Sep 16 19:06:45.840067 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.840043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwzlf"] Sep 16 19:06:45.888540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:45.888540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-data-volume\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.888659 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.888678 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.888729 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.888714575 +0000 UTC m=+94.094670310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-crio-socket\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.888772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.888761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2c4f\" (UniqueName: \"kubernetes.io/projected/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-api-access-w2c4f\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.989854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.989813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-crio-socket\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.989854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.989857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2c4f\" (UniqueName: \"kubernetes.io/projected/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-api-access-w2c4f\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.990163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.989947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-crio-socket\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.990163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.989996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.990163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.990024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-data-volume\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.990163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.990047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.990163 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.990139 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:45.990436 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.990204 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.490184289 +0000 UTC m=+90.696140034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:45.990436 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.990139 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:45.990436 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:45.990317 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:46.490275265 +0000 UTC m=+90.696231010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : configmap "kube-rbac-proxy" not found Sep 16 19:06:45.990436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.990418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-data-volume\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:45.999599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:45.999578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2c4f\" (UniqueName: \"kubernetes.io/projected/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-api-access-w2c4f\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:46.494098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:46.494059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:46.494343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:46.494107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:46.494343 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.494179 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:46.494343 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.494223 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:46.494343 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.494252 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.494235089 +0000 UTC m=+91.700190839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : configmap "kube-rbac-proxy" not found Sep 16 19:06:46.494343 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.494300 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:47.494265539 +0000 UTC m=+91.700221269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:46.796603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:46.796499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:46.796603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:46.796575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:46.797051 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.796674 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:50.79665641 +0000 UTC m=+95.002612145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : configmap references non-existent config key: service-ca.crt Sep 16 19:06:46.797051 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.796716 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 19:06:46.797051 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:46.796779 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:50.796763744 +0000 UTC m=+95.002719494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : secret "router-metrics-certs-default" not found Sep 16 19:06:47.502113 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:47.502072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:47.502346 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:47.502181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:47.502346 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:47.502203 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:47.502346 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:47.502275 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:47.502346 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:47.502296 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.502264902 +0000 UTC m=+93.708220638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : configmap "kube-rbac-proxy" not found Sep 16 19:06:47.502500 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:47.502352 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:49.502340742 +0000 UTC m=+93.708296472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:47.708794 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:47.708762 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fwsh_b547de82-20e4-4a98-b580-70f04676b620/dns-node-resolver/0.log" Sep 16 19:06:48.912908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:48.912879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj67g_297542fb-6cc8-409f-a483-4749340921ef/node-ca/0.log" Sep 16 19:06:49.516523 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:49.516490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:49.516712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:49.516531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:49.516712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.516632 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:49.516712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.516645 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:49.516712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.516695 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.516681226 +0000 UTC m=+97.722636957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : configmap "kube-rbac-proxy" not found Sep 16 19:06:49.516712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.516708 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:53.516702437 +0000 UTC m=+97.722658168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:49.710772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:49.710736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wvl7c_18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b/serve-healthcheck-canary/0.log" Sep 16 19:06:49.920733 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:49.920655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:49.921062 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.920808 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:49.921062 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.920828 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:49.921062 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:49.920885 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:06:57.920868242 +0000 UTC m=+102.126823997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:50.826903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:50.826859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:50.827089 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:50.826916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:50.827089 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:50.827025 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 19:06:50.827089 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:50.827032 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:58.827014598 +0000 UTC m=+103.032970334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : configmap references non-existent config key: service-ca.crt Sep 16 19:06:50.827089 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:50.827078 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:58.82706197 +0000 UTC m=+103.033017701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : secret "router-metrics-certs-default" not found Sep 16 19:06:53.490655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.490614 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nrw2t"] Sep 16 19:06:53.494071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.494049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.496812 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.496792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 16 19:06:53.497932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.497914 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 16 19:06:53.498006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.497988 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jnjgm\"" Sep 16 19:06:53.498054 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.498036 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 16 19:06:53.498377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.498359 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 16 19:06:53.504442 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.504421 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nrw2t"] Sep 16 19:06:53.547396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.547351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-key\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.547581 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.547403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-cabundle\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.547581 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.547531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:53.547581 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.547574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:06:53.547739 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.547604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hq4\" (UniqueName: \"kubernetes.io/projected/9b350f9c-9ab1-4781-b27c-894188a19f83-kube-api-access-c2hq4\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.547739 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:53.547675 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 19:06:53.547739 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:53.547689 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:06:53.547739 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:53.547728 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.547714085 +0000 UTC m=+105.753669822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : configmap "kube-rbac-proxy" not found Sep 16 19:06:53.547880 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:53.547755 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:07:01.547736641 +0000 UTC m=+105.753692373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:06:53.648385 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.648346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hq4\" (UniqueName: \"kubernetes.io/projected/9b350f9c-9ab1-4781-b27c-894188a19f83-kube-api-access-c2hq4\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.648556 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.648410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-key\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.648556 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.648432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-cabundle\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.649113 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.649045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-cabundle\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.651044 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.651018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b350f9c-9ab1-4781-b27c-894188a19f83-signing-key\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.659633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.659604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hq4\" (UniqueName: \"kubernetes.io/projected/9b350f9c-9ab1-4781-b27c-894188a19f83-kube-api-access-c2hq4\") pod \"service-ca-568869fdc5-nrw2t\" (UID: \"9b350f9c-9ab1-4781-b27c-894188a19f83\") " pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:53.803500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:53.803386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" Sep 16 19:06:54.074112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:54.074089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nrw2t"] Sep 16 19:06:54.076340 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:06:54.076312 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b350f9c_9ab1_4781_b27c_894188a19f83.slice/crio-5a1098deb9e64e8d3e4fb4a6e9d70abee9b9649a35b4c4eca68f580935218dfe WatchSource:0}: Error finding container 5a1098deb9e64e8d3e4fb4a6e9d70abee9b9649a35b4c4eca68f580935218dfe: Status 404 returned error can't find the container with id 5a1098deb9e64e8d3e4fb4a6e9d70abee9b9649a35b4c4eca68f580935218dfe Sep 16 19:06:54.596105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:54.596069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" event={"ID":"9b350f9c-9ab1-4781-b27c-894188a19f83","Type":"ContainerStarted","Data":"5a1098deb9e64e8d3e4fb4a6e9d70abee9b9649a35b4c4eca68f580935218dfe"} Sep 16 19:06:55.161885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:55.161851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:06:55.162058 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:55.161989 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 19:06:55.162058 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:55.162055 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls podName:8452df11-18f3-46c0-9353-920103de5013 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:59.162036849 +0000 UTC m=+163.367992580 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls") pod "dns-default-969r4" (UID: "8452df11-18f3-46c0-9353-920103de5013") : secret "dns-default-metrics-tls" not found Sep 16 19:06:56.604366 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:56.604323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" event={"ID":"9b350f9c-9ab1-4781-b27c-894188a19f83","Type":"ContainerStarted","Data":"1e98fb06b04468515453b0c09e6452b8383f92ae7893fc913dd3f61a4c6382aa"} Sep 16 19:06:56.624750 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:56.624696 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-nrw2t" podStartSLOduration=1.780404011 podStartE2EDuration="3.624678708s" totalImagesPullingTime="1.844274697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:56.623523672 +0000 UTC m=+100.829479424" watchObservedRunningTime="2026-09-16 19:06:56.624678708 +0000 UTC m=+100.830634461" Sep 16 19:06:57.984054 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:57.984008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:06:57.984568 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:57.984178 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 19:06:57.984568 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:57.984206 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-676654c77c-4559p: secret "image-registry-tls" not found Sep 16 19:06:57.990301 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:57.984850 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls podName:566a56a7-ea6b-46a2-8da1-01325ba9c9af nodeName:}" failed. No retries permitted until 2026-09-16 19:07:13.984723015 +0000 UTC m=+118.190679072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls") pod "image-registry-676654c77c-4559p" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af") : secret "image-registry-tls" not found Sep 16 19:06:58.524593 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:58.524566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kg6ff" Sep 16 19:06:58.893189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:58.893096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:58.893365 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:06:58.893215 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:06:58.893365 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:58.893252 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 19:06:58.893365 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:58.893357 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:14.893338587 +0000 UTC m=+119.099294332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : configmap references non-existent config key: service-ca.crt Sep 16 19:06:58.893528 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:06:58.893383 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs podName:70722144-5165-41ed-9e93-0c3fd3354607 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:14.893371805 +0000 UTC m=+119.099327539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs") pod "router-default-6cf589d94b-rxcqc" (UID: "70722144-5165-41ed-9e93-0c3fd3354607") : secret "router-metrics-certs-default" not found Sep 16 19:07:01.614300 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:01.614242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:01.614784 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:01.614420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:01.614784 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:07:01.614553 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 19:07:01.614784 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:07:01.614635 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls podName:9dd12d57-2add-45a4-9acd-ceb8d543cb0d nodeName:}" failed. No retries permitted until 2026-09-16 19:07:17.61461513 +0000 UTC m=+121.820570865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwzlf" (UID: "9dd12d57-2add-45a4-9acd-ceb8d543cb0d") : secret "insights-runtime-extractor-tls" not found Sep 16 19:07:01.614948 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:01.614795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:14.015245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.015203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:14.017866 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.017842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"image-registry-676654c77c-4559p\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:14.052041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.052010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:14.359171 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.359086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:07:14.362337 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:14.362308 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566a56a7_ea6b_46a2_8da1_01325ba9c9af.slice/crio-2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e WatchSource:0}: Error finding container 2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e: Status 404 returned error can't find the container with id 2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e Sep 16 19:07:14.659067 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.658983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676654c77c-4559p" event={"ID":"566a56a7-ea6b-46a2-8da1-01325ba9c9af","Type":"ContainerStarted","Data":"2852d0828a527c1e3d038f50ff0e801d4e48ead05f217caacb759d27f9a67447"} Sep 16 19:07:14.659067 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.659029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676654c77c-4559p" event={"ID":"566a56a7-ea6b-46a2-8da1-01325ba9c9af","Type":"ContainerStarted","Data":"2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e"} Sep 16 19:07:14.659067 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.659060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:14.691016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.690956 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-676654c77c-4559p" podStartSLOduration=32.690939607 podStartE2EDuration="32.690939607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:14.690263751 +0000 UTC m=+118.896219506" watchObservedRunningTime="2026-09-16 19:07:14.690939607 +0000 UTC m=+118.896895400" Sep 16 19:07:14.928978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.928870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:14.928978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.928942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:14.929540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.929515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70722144-5165-41ed-9e93-0c3fd3354607-service-ca-bundle\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:14.931386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:14.931361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70722144-5165-41ed-9e93-0c3fd3354607-metrics-certs\") pod \"router-default-6cf589d94b-rxcqc\" (UID: \"70722144-5165-41ed-9e93-0c3fd3354607\") " pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:15.144963 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:15.144930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:15.424181 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:15.424156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cf589d94b-rxcqc"] Sep 16 19:07:15.426428 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:15.426401 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70722144_5165_41ed_9e93_0c3fd3354607.slice/crio-9d65a596a732778ff318e304216e2685e0d690ce180fc375c3f424b5f32edee6 WatchSource:0}: Error finding container 9d65a596a732778ff318e304216e2685e0d690ce180fc375c3f424b5f32edee6: Status 404 returned error can't find the container with id 9d65a596a732778ff318e304216e2685e0d690ce180fc375c3f424b5f32edee6 Sep 16 19:07:15.663320 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:15.663270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" event={"ID":"70722144-5165-41ed-9e93-0c3fd3354607","Type":"ContainerStarted","Data":"e61390a4ccd10a96ec3c32fafa85e9714961386c157f7b803ea10776d6bdb83f"} Sep 16 19:07:15.663487 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:15.663342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" event={"ID":"70722144-5165-41ed-9e93-0c3fd3354607","Type":"ContainerStarted","Data":"9d65a596a732778ff318e304216e2685e0d690ce180fc375c3f424b5f32edee6"} Sep 16 19:07:15.700715 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:15.700664 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" podStartSLOduration=33.700648309 podStartE2EDuration="33.700648309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:15.698958866 +0000 UTC m=+119.904914620" watchObservedRunningTime="2026-09-16 19:07:15.700648309 +0000 UTC m=+119.906604062" Sep 16 19:07:16.146044 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:16.146001 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:16.148550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:16.148525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:16.666096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:16.666070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:16.667305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:16.667261 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cf589d94b-rxcqc" Sep 16 19:07:17.649496 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:17.649446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:17.651998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:17.651979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dd12d57-2add-45a4-9acd-ceb8d543cb0d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwzlf\" (UID: \"9dd12d57-2add-45a4-9acd-ceb8d543cb0d\") " pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:17.933910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:17.933711 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-27w29\"" Sep 16 19:07:17.941599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:17.941576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwzlf" Sep 16 19:07:18.213408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.213327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwzlf"] Sep 16 19:07:18.216728 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:18.216699 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd12d57_2add_45a4_9acd_ceb8d543cb0d.slice/crio-59aa177d433d70c73f341a3f7f00f7f2649bdc56ec6f5c58185b2a167c257cff WatchSource:0}: Error finding container 59aa177d433d70c73f341a3f7f00f7f2649bdc56ec6f5c58185b2a167c257cff: Status 404 returned error can't find the container with id 59aa177d433d70c73f341a3f7f00f7f2649bdc56ec6f5c58185b2a167c257cff Sep 16 19:07:18.515628 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.515593 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq"] Sep 16 19:07:18.521689 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.520782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.526536 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.526513 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 19:07:18.526677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.526628 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 19:07:18.526677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.526636 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-97krg\"" Sep 16 19:07:18.539992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.539968 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq"] Sep 16 19:07:18.551157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.551130 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:07:18.658515 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.658475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2272dae0-6797-4da6-a872-9b1e8fc7711c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.658915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.658558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2272dae0-6797-4da6-a872-9b1e8fc7711c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.673658 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.673624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwzlf" event={"ID":"9dd12d57-2add-45a4-9acd-ceb8d543cb0d","Type":"ContainerStarted","Data":"02e69295b66df9f2eb2824a9d6d024691e6be05f80d3fd4aa5852be39521617c"} Sep 16 19:07:18.673792 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.673684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwzlf" event={"ID":"9dd12d57-2add-45a4-9acd-ceb8d543cb0d","Type":"ContainerStarted","Data":"59aa177d433d70c73f341a3f7f00f7f2649bdc56ec6f5c58185b2a167c257cff"} Sep 16 19:07:18.759207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.759172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2272dae0-6797-4da6-a872-9b1e8fc7711c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.759500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.759472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2272dae0-6797-4da6-a872-9b1e8fc7711c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.759958 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.759937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2272dae0-6797-4da6-a872-9b1e8fc7711c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.761988 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.761971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2272dae0-6797-4da6-a872-9b1e8fc7711c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-dpzbq\" (UID: \"2272dae0-6797-4da6-a872-9b1e8fc7711c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:18.830305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:18.830199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" Sep 16 19:07:19.200412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:19.200389 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq"] Sep 16 19:07:19.201910 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:19.201882 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2272dae0_6797_4da6_a872_9b1e8fc7711c.slice/crio-50aa80788ff6a1400f6541ed381f32cd59fe4ed61255470d3e66172f957cc068 WatchSource:0}: Error finding container 50aa80788ff6a1400f6541ed381f32cd59fe4ed61255470d3e66172f957cc068: Status 404 returned error can't find the container with id 50aa80788ff6a1400f6541ed381f32cd59fe4ed61255470d3e66172f957cc068 Sep 16 19:07:19.679858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:19.679822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwzlf" event={"ID":"9dd12d57-2add-45a4-9acd-ceb8d543cb0d","Type":"ContainerStarted","Data":"1bc12f4458439b4e13adb04626e60e29b7fc43910a8bcb32b02d1689e12f44a2"} Sep 16 19:07:19.681470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:19.681447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" event={"ID":"2272dae0-6797-4da6-a872-9b1e8fc7711c","Type":"ContainerStarted","Data":"50aa80788ff6a1400f6541ed381f32cd59fe4ed61255470d3e66172f957cc068"} Sep 16 19:07:20.686994 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:20.686960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwzlf" event={"ID":"9dd12d57-2add-45a4-9acd-ceb8d543cb0d","Type":"ContainerStarted","Data":"8e0dcf327804b1bd64b710a2c508179e5cde6dd0b891a4793381c9698a99486a"} Sep 16 19:07:20.688563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:20.688540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" event={"ID":"2272dae0-6797-4da6-a872-9b1e8fc7711c","Type":"ContainerStarted","Data":"8f356ae5a63e31d883678323ea7e093452f2eaf32c05fe5bbd200d7d3f19e500"} Sep 16 19:07:20.708873 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:20.708826 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mwzlf" podStartSLOduration=33.443578743 podStartE2EDuration="35.708810538s" totalImagesPullingTime="2.265231795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:06:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:20.70764034 +0000 UTC m=+124.913596092" watchObservedRunningTime="2026-09-16 19:07:20.708810538 +0000 UTC m=+124.914766291" Sep 16 19:07:20.734940 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:20.734890 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-dpzbq" podStartSLOduration=1.332455295 podStartE2EDuration="2.734876538s" totalImagesPullingTime="1.402421243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:20.733741651 +0000 UTC m=+124.939697404" watchObservedRunningTime="2026-09-16 19:07:20.734876538 +0000 UTC m=+124.940832284" Sep 16 19:07:23.131615 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.131581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm"] Sep 16 19:07:23.135786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.135769 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.153084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153051 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 19:07:23.153207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153086 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 19:07:23.153853 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153836 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 19:07:23.153853 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153839 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 19:07:23.153986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153842 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 19:07:23.154039 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.153991 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mgsqq\"" Sep 16 19:07:23.171956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.171931 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm"] Sep 16 19:07:23.294623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.294590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.294623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.294631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mrn\" (UniqueName: \"kubernetes.io/projected/b1dad97e-78e6-4f43-9f82-19335c54019d-kube-api-access-j2mrn\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.294879 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.294671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dad97e-78e6-4f43-9f82-19335c54019d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.294879 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.294700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.396008 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.395925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.396008 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.395977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mrn\" (UniqueName: \"kubernetes.io/projected/b1dad97e-78e6-4f43-9f82-19335c54019d-kube-api-access-j2mrn\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.396217 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.396017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dad97e-78e6-4f43-9f82-19335c54019d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.396217 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.396046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.396718 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.396701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dad97e-78e6-4f43-9f82-19335c54019d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.398321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.398264 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.398417 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.398401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dad97e-78e6-4f43-9f82-19335c54019d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.415754 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.415731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mrn\" (UniqueName: \"kubernetes.io/projected/b1dad97e-78e6-4f43-9f82-19335c54019d-kube-api-access-j2mrn\") pod \"prometheus-operator-68bcd7f987-mxzsm\" (UID: \"b1dad97e-78e6-4f43-9f82-19335c54019d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.444766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.444740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" Sep 16 19:07:23.730142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:23.730116 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm"] Sep 16 19:07:23.731873 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:23.731851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dad97e_78e6_4f43_9f82_19335c54019d.slice/crio-245ec7633afa460a5262547a37f3b32d72810635f3039d0c52cb6d9e184d613d WatchSource:0}: Error finding container 245ec7633afa460a5262547a37f3b32d72810635f3039d0c52cb6d9e184d613d: Status 404 returned error can't find the container with id 245ec7633afa460a5262547a37f3b32d72810635f3039d0c52cb6d9e184d613d Sep 16 19:07:24.702027 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:24.701988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" event={"ID":"b1dad97e-78e6-4f43-9f82-19335c54019d","Type":"ContainerStarted","Data":"245ec7633afa460a5262547a37f3b32d72810635f3039d0c52cb6d9e184d613d"} Sep 16 19:07:25.706915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:25.706880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" event={"ID":"b1dad97e-78e6-4f43-9f82-19335c54019d","Type":"ContainerStarted","Data":"3e5b0955733c6953351c2f144fa82073f4bf9ef1ac51a04cc97a6876851eb3d4"} Sep 16 19:07:25.707325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:25.706930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" event={"ID":"b1dad97e-78e6-4f43-9f82-19335c54019d","Type":"ContainerStarted","Data":"69a50f1cfffdafb9c68be25b5c259ed8166b838a8fbf4f1a9754d43d23b1458d"} Sep 16 19:07:25.745381 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:25.745331 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mxzsm" podStartSLOduration=1.4801892859999999 podStartE2EDuration="2.745315741s" totalImagesPullingTime="1.265126455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:25.745218963 +0000 UTC m=+129.951174715" watchObservedRunningTime="2026-09-16 19:07:25.745315741 +0000 UTC m=+129.951271494" Sep 16 19:07:26.016991 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.016954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:07:26.019453 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.019430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4ea1abe-95f7-4dbc-b130-c3b9b471452a-metrics-certs\") pod \"network-metrics-daemon-vnwn8\" (UID: \"b4ea1abe-95f7-4dbc-b130-c3b9b471452a\") " pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:07:26.166343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.166309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjb6w\"" Sep 16 19:07:26.174505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.174484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vnwn8" Sep 16 19:07:26.447003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.446827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vnwn8"] Sep 16 19:07:26.449549 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:26.449520 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea1abe_95f7_4dbc_b130_c3b9b471452a.slice/crio-1b071b9e4b2bc2c016e68d8136e747eb4636ead71ac167d64c666f87c87d642b WatchSource:0}: Error finding container 1b071b9e4b2bc2c016e68d8136e747eb4636ead71ac167d64c666f87c87d642b: Status 404 returned error can't find the container with id 1b071b9e4b2bc2c016e68d8136e747eb4636ead71ac167d64c666f87c87d642b Sep 16 19:07:26.711902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:26.711804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnwn8" event={"ID":"b4ea1abe-95f7-4dbc-b130-c3b9b471452a","Type":"ContainerStarted","Data":"1b071b9e4b2bc2c016e68d8136e747eb4636ead71ac167d64c666f87c87d642b"} Sep 16 19:07:27.629111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.629032 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl"] Sep 16 19:07:27.632611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.632591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.635331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.635305 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 19:07:27.635707 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.635658 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9l5gp\"" Sep 16 19:07:27.635801 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.635730 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 19:07:27.643013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.642532 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzw2w"] Sep 16 19:07:27.645972 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.645952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.648715 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.648692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 19:07:27.649155 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.649013 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 19:07:27.649155 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.649026 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rp57\"" Sep 16 19:07:27.649982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.649962 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 19:07:27.651210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.651190 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6"] Sep 16 19:07:27.654929 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.654904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl"] Sep 16 19:07:27.654929 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.654928 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.660367 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.660349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 19:07:27.661037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.661017 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 19:07:27.662010 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.661807 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xnsx8\"" Sep 16 19:07:27.664160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.664142 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 19:07:27.695612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.695585 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6"] Sep 16 19:07:27.718365 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.718200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnwn8" event={"ID":"b4ea1abe-95f7-4dbc-b130-c3b9b471452a","Type":"ContainerStarted","Data":"68bc9354a5f91af8e49911cb15d6f8a979fd38e3493068f3d1c67a5d916316e2"} Sep 16 19:07:27.718365 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.718266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vnwn8" event={"ID":"b4ea1abe-95f7-4dbc-b130-c3b9b471452a","Type":"ContainerStarted","Data":"6fda9dff42e722a3d45a8bebdbb3cd9a00cbc0c4589d2069f3fac5da0907c724"} Sep 16 19:07:27.731871 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.731844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2p6\" (UniqueName: \"kubernetes.io/projected/c23a1749-534f-490b-8acf-74223843a16f-kube-api-access-mp2p6\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.732038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.731887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.732038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.731942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23a1749-534f-490b-8acf-74223843a16f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.732038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.731971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.819466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.819403 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vnwn8" podStartSLOduration=130.896658918 podStartE2EDuration="2m11.819383238s" totalImagesPullingTime="922.72432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:27.817727375 +0000 UTC m=+132.023683128" watchObservedRunningTime="2026-09-16 19:07:27.819383238 +0000 UTC m=+132.025338992" Sep 16 19:07:27.832749 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.832910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-sys\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.832910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-metrics-client-ca\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.832910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbvd\" (UniqueName: \"kubernetes.io/projected/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-api-access-fcbvd\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.833034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-textfile\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.832985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplr4\" (UniqueName: \"kubernetes.io/projected/40643aef-5885-4b34-9d3c-3e3b3e34037d-kube-api-access-dplr4\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-wtmp\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2p6\" (UniqueName: \"kubernetes.io/projected/c23a1749-534f-490b-8acf-74223843a16f-kube-api-access-mp2p6\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-root\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.833206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-tls\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.833539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.833539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23a1749-534f-490b-8acf-74223843a16f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.833539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.833400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.834047 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.834024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23a1749-534f-490b-8acf-74223843a16f-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.835641 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.835623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.835929 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.835907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c23a1749-534f-490b-8acf-74223843a16f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.848870 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.848843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2p6\" (UniqueName: \"kubernetes.io/projected/c23a1749-534f-490b-8acf-74223843a16f-kube-api-access-mp2p6\") pod \"openshift-state-metrics-67dcf8c979-tw8vl\" (UID: \"c23a1749-534f-490b-8acf-74223843a16f\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.933859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.933775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-sys\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.933859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.933812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-metrics-client-ca\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.933859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.933829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbvd\" (UniqueName: \"kubernetes.io/projected/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-api-access-fcbvd\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.933905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-sys\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-textfile\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplr4\" (UniqueName: \"kubernetes.io/projected/40643aef-5885-4b34-9d3c-3e3b3e34037d-kube-api-access-dplr4\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934113 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-wtmp\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-root\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-tls\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-wtmp\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-textfile\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.934828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-metrics-client-ca\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.934828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.934791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.935104 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.935078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40643aef-5885-4b34-9d3c-3e3b3e34037d-root\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.935378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.935206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.935378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.935346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.937739 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.937713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.937824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.937795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.938024 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.938005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.938084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.938069 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40643aef-5885-4b34-9d3c-3e3b3e34037d-node-exporter-tls\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.942819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.942801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" Sep 16 19:07:27.946806 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.946781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplr4\" (UniqueName: \"kubernetes.io/projected/40643aef-5885-4b34-9d3c-3e3b3e34037d-kube-api-access-dplr4\") pod \"node-exporter-mzw2w\" (UID: \"40643aef-5885-4b34-9d3c-3e3b3e34037d\") " pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.948242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.948217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbvd\" (UniqueName: \"kubernetes.io/projected/2537aaa2-0a6f-4c57-875b-aeb59cbf5555-kube-api-access-fcbvd\") pod \"kube-state-metrics-59fbdd77d-mxbx6\" (UID: \"2537aaa2-0a6f-4c57-875b-aeb59cbf5555\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.956716 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.956694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzw2w" Sep 16 19:07:27.965121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:27.965086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" Sep 16 19:07:27.970462 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:27.970435 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40643aef_5885_4b34_9d3c_3e3b3e34037d.slice/crio-0f3301c22f1377df5f073a70de9f2e16c920763d2bf0d7e1564608a030b9dde3 WatchSource:0}: Error finding container 0f3301c22f1377df5f073a70de9f2e16c920763d2bf0d7e1564608a030b9dde3: Status 404 returned error can't find the container with id 0f3301c22f1377df5f073a70de9f2e16c920763d2bf0d7e1564608a030b9dde3 Sep 16 19:07:28.237341 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.237318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl"] Sep 16 19:07:28.239267 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:28.239246 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23a1749_534f_490b_8acf_74223843a16f.slice/crio-71fcaddd17d70c3232801024562aa9c16b70abc40fcdb5cd5157fa6657ee3ca2 WatchSource:0}: Error finding container 71fcaddd17d70c3232801024562aa9c16b70abc40fcdb5cd5157fa6657ee3ca2: Status 404 returned error can't find the container with id 71fcaddd17d70c3232801024562aa9c16b70abc40fcdb5cd5157fa6657ee3ca2 Sep 16 19:07:28.271741 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.271707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6"] Sep 16 19:07:28.275451 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:28.275424 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2537aaa2_0a6f_4c57_875b_aeb59cbf5555.slice/crio-5e05f5646b8630ff725434ce20fecbdc4d34554d838a19a992682709fd07e88b WatchSource:0}: Error finding container 5e05f5646b8630ff725434ce20fecbdc4d34554d838a19a992682709fd07e88b: Status 404 returned error can't find the container with id 5e05f5646b8630ff725434ce20fecbdc4d34554d838a19a992682709fd07e88b Sep 16 19:07:28.724703 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.724659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" event={"ID":"c23a1749-534f-490b-8acf-74223843a16f","Type":"ContainerStarted","Data":"410cfea64ebfc63ee94ba83be378515a39cfd83c728f2b6f134d00b038e75375"} Sep 16 19:07:28.725105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.724735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" event={"ID":"c23a1749-534f-490b-8acf-74223843a16f","Type":"ContainerStarted","Data":"4b2ea5dee4f478b727fc8310650b12b8e4e915b6239d8fa696267c33d8a495ee"} Sep 16 19:07:28.725105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.724756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" event={"ID":"c23a1749-534f-490b-8acf-74223843a16f","Type":"ContainerStarted","Data":"71fcaddd17d70c3232801024562aa9c16b70abc40fcdb5cd5157fa6657ee3ca2"} Sep 16 19:07:28.725982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.725952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" event={"ID":"2537aaa2-0a6f-4c57-875b-aeb59cbf5555","Type":"ContainerStarted","Data":"5e05f5646b8630ff725434ce20fecbdc4d34554d838a19a992682709fd07e88b"} Sep 16 19:07:28.727127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:28.727105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzw2w" event={"ID":"40643aef-5885-4b34-9d3c-3e3b3e34037d","Type":"ContainerStarted","Data":"0f3301c22f1377df5f073a70de9f2e16c920763d2bf0d7e1564608a030b9dde3"} Sep 16 19:07:29.734044 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:29.733988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" event={"ID":"c23a1749-534f-490b-8acf-74223843a16f","Type":"ContainerStarted","Data":"ee8513d99250f8dc809ea99c0a9c6c27ef189feee85d996136984402199d5f0e"} Sep 16 19:07:29.737149 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:29.737114 2565 generic.go:356] "Generic (PLEG): container finished" podID="40643aef-5885-4b34-9d3c-3e3b3e34037d" containerID="e659c79d8d1952d32b8d2b8ad5ee772bb3f876cee163a12dbfeb0eda465900fe" exitCode=0 Sep 16 19:07:29.737305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:29.737167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzw2w" event={"ID":"40643aef-5885-4b34-9d3c-3e3b3e34037d","Type":"ContainerDied","Data":"e659c79d8d1952d32b8d2b8ad5ee772bb3f876cee163a12dbfeb0eda465900fe"} Sep 16 19:07:29.778882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:29.778763 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw8vl" podStartSLOduration=1.644555056 podStartE2EDuration="2.778742616s" totalImagesPullingTime="1.13418756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:29.777697112 +0000 UTC m=+133.983652866" watchObservedRunningTime="2026-09-16 19:07:29.778742616 +0000 UTC m=+133.984698372" Sep 16 19:07:30.750049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.745720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" event={"ID":"2537aaa2-0a6f-4c57-875b-aeb59cbf5555","Type":"ContainerStarted","Data":"df5cf5aae3ca20449e3a8b16ad241cf59bf3592ca4482301c30f378c4f5490c4"} Sep 16 19:07:30.750049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.745830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" event={"ID":"2537aaa2-0a6f-4c57-875b-aeb59cbf5555","Type":"ContainerStarted","Data":"d0abc89a1d6a5e297351a4d92ec099b1fb7a9c3c509ea99dbb324b626876ac03"} Sep 16 19:07:30.750049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.745873 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8568f88679-mblvk"] Sep 16 19:07:30.754518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.754494 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" event={"ID":"2537aaa2-0a6f-4c57-875b-aeb59cbf5555","Type":"ContainerStarted","Data":"0159c8d769b10e15ef4340ad81aa3219eb13b3cc143f3f23006ed655227d8232"} Sep 16 19:07:30.754641 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.754526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzw2w" event={"ID":"40643aef-5885-4b34-9d3c-3e3b3e34037d","Type":"ContainerStarted","Data":"55810d011000c36b2889174793d552db872726a4cdb418b0498637410c230b70"} Sep 16 19:07:30.754641 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.754550 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzw2w" event={"ID":"40643aef-5885-4b34-9d3c-3e3b3e34037d","Type":"ContainerStarted","Data":"50923d764b6bd0963664e5f521812984ccb3891e445873d1c2a16e0d06248e52"} Sep 16 19:07:30.754641 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.754629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.763508 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.763485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 19:07:30.763622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.763604 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 19:07:30.763845 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.763830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e15j630918sr3\"" Sep 16 19:07:30.763981 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.763965 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 19:07:30.764036 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.763989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 19:07:30.764268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.764249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6jp79\"" Sep 16 19:07:30.764393 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.764351 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 19:07:30.795611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.795557 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-mxbx6" podStartSLOduration=2.156156319 podStartE2EDuration="3.795536508s" totalImagesPullingTime="1.639380189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:30.785691519 +0000 UTC m=+134.991647272" watchObservedRunningTime="2026-09-16 19:07:30.795536508 +0000 UTC m=+135.001492263" Sep 16 19:07:30.796123 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.796104 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8568f88679-mblvk"] Sep 16 19:07:30.861935 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.861903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-metrics-client-ca\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862093 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.861972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-grpc-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862093 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862093 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862254 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862508 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.862599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.862564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f7d\" (UniqueName: \"kubernetes.io/projected/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-kube-api-access-49f7d\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.963671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-metrics-client-ca\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.963671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-grpc-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.963902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.963902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.963902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.964065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.964065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.963980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.964065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.964009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f7d\" (UniqueName: \"kubernetes.io/projected/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-kube-api-access-49f7d\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.964833 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.964788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-metrics-client-ca\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.966776 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.966734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-grpc-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.966927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.966905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.967242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.967219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.967335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.967314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.967379 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.967359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.967625 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.967609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-secret-thanos-querier-tls\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:30.973553 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:30.973532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f7d\" (UniqueName: \"kubernetes.io/projected/fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe-kube-api-access-49f7d\") pod \"thanos-querier-8568f88679-mblvk\" (UID: \"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe\") " pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:31.064108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:31.064015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:31.374652 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:31.374545 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mzw2w" podStartSLOduration=3.305423454 podStartE2EDuration="4.374526494s" totalImagesPullingTime="990.112272ms" totalInitContainerRuntime="78.990768ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:30.868092535 +0000 UTC m=+135.074048313" watchObservedRunningTime="2026-09-16 19:07:31.374526494 +0000 UTC m=+135.580482247" Sep 16 19:07:31.375359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:31.375322 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8568f88679-mblvk"] Sep 16 19:07:31.377573 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:31.377551 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3a7fa7_a423_4f4e_bc2f_8f3488460dfe.slice/crio-340fdc51c48503b27eb9273011bcabe73289416cdd8c152cfe952463af6490dc WatchSource:0}: Error finding container 340fdc51c48503b27eb9273011bcabe73289416cdd8c152cfe952463af6490dc: Status 404 returned error can't find the container with id 340fdc51c48503b27eb9273011bcabe73289416cdd8c152cfe952463af6490dc Sep 16 19:07:31.753762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:31.753727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"340fdc51c48503b27eb9273011bcabe73289416cdd8c152cfe952463af6490dc"} Sep 16 19:07:33.768653 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:33.768615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"e06ab4026a121c4e55eeaa64128d6eac450992df14b07e6f2871cdab30efea8c"} Sep 16 19:07:33.769023 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:33.768705 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"978db05a29c77add0b6a352f13a5026f1d41409bcd6005f5bf7ed6bdea6fe224"} Sep 16 19:07:33.769023 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:33.768730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"42224be4e9f864a978d05d145d3b79f60de3b97e28250313ad9724aec4a5996b"} Sep 16 19:07:34.776242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:34.776202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"0f16a899ae8910629105f733e65d89d81300a31a5724c4376801bbc2da409b3d"} Sep 16 19:07:34.776695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:34.776272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"a473ffdaeb8701c30a988c77d5390cbdb05e2b2fe00eed2b8394af7b38784fe3"} Sep 16 19:07:34.776695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:34.776330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" event={"ID":"fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe","Type":"ContainerStarted","Data":"b6452a0d53f21e8adad0182da074df0fbd587b5195c5ac870bc1d33da2c9630e"} Sep 16 19:07:34.776695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:34.776367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:34.809907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:34.809837 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" podStartSLOduration=2.314607998 podStartE2EDuration="4.809817637s" totalImagesPullingTime="2.495209639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:34.807798239 +0000 UTC m=+139.013754004" watchObservedRunningTime="2026-09-16 19:07:34.809817637 +0000 UTC m=+139.015773391" Sep 16 19:07:38.556703 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:38.556660 2565 patch_prober.go:28] interesting pod/image-registry-676654c77c-4559p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 19:07:38.557083 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:38.556714 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-676654c77c-4559p" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:07:40.786903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:40.786873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8568f88679-mblvk" Sep 16 19:07:43.571982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.571912 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-676654c77c-4559p" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerName="registry" containerID="cri-o://2852d0828a527c1e3d038f50ff0e801d4e48ead05f217caacb759d27f9a67447" gracePeriod=30 Sep 16 19:07:43.813813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.813783 2565 generic.go:356] "Generic (PLEG): container finished" podID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerID="2852d0828a527c1e3d038f50ff0e801d4e48ead05f217caacb759d27f9a67447" exitCode=0 Sep 16 19:07:43.813942 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.813816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676654c77c-4559p" event={"ID":"566a56a7-ea6b-46a2-8da1-01325ba9c9af","Type":"ContainerDied","Data":"2852d0828a527c1e3d038f50ff0e801d4e48ead05f217caacb759d27f9a67447"} Sep 16 19:07:43.813942 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.813875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676654c77c-4559p" event={"ID":"566a56a7-ea6b-46a2-8da1-01325ba9c9af","Type":"ContainerDied","Data":"2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e"} Sep 16 19:07:43.813942 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.813897 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb53a78bc3813fbbacfc9ae31ba3b4d6438d2145ad56060079ec89479279e4e" Sep 16 19:07:43.816180 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.816165 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:43.865786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qxqc\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.865786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865756 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.865998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.865998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865902 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.865998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865955 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.866133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.865998 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.866133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.866059 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.866133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.866102 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca\") pod \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\" (UID: \"566a56a7-ea6b-46a2-8da1-01325ba9c9af\") " Sep 16 19:07:43.866327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.866242 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:43.866520 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.866397 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-certificates\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.866654 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.866633 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:43.868565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.868435 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:43.868565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.868537 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc" (OuterVolumeSpecName: "kube-api-access-8qxqc") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "kube-api-access-8qxqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:43.868724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.868682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:43.868778 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.868718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:43.868778 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.868726 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:43.874486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.874462 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "566a56a7-ea6b-46a2-8da1-01325ba9c9af" (UID: "566a56a7-ea6b-46a2-8da1-01325ba9c9af"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:07:43.966946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966911 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxqc\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-kube-api-access-8qxqc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.966946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966940 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-bound-sa-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.966946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966951 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-image-registry-private-configuration\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.967168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966960 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/566a56a7-ea6b-46a2-8da1-01325ba9c9af-ca-trust-extracted\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.967168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966969 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/566a56a7-ea6b-46a2-8da1-01325ba9c9af-installation-pull-secrets\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.967168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966978 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/566a56a7-ea6b-46a2-8da1-01325ba9c9af-registry-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.967168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.966985 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/566a56a7-ea6b-46a2-8da1-01325ba9c9af-trusted-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:07:43.991092 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.991065 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-ck5md"] Sep 16 19:07:43.991390 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.991377 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerName="registry" Sep 16 19:07:43.991438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.991392 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerName="registry" Sep 16 19:07:43.991468 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.991451 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" containerName="registry" Sep 16 19:07:43.996038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:43.996019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:07:44.000816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.000790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bqjdt\"" Sep 16 19:07:44.000908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.000877 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 19:07:44.000908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.000883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 19:07:44.014577 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.014550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ck5md"] Sep 16 19:07:44.068057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.068026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4cw\" (UniqueName: \"kubernetes.io/projected/56b360ba-98b8-4d60-b833-9a2d3f842a0d-kube-api-access-nr4cw\") pod \"downloads-76f99584f4-ck5md\" (UID: \"56b360ba-98b8-4d60-b833-9a2d3f842a0d\") " pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:07:44.168967 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.168883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4cw\" (UniqueName: \"kubernetes.io/projected/56b360ba-98b8-4d60-b833-9a2d3f842a0d-kube-api-access-nr4cw\") pod \"downloads-76f99584f4-ck5md\" (UID: \"56b360ba-98b8-4d60-b833-9a2d3f842a0d\") " pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:07:44.180643 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.180615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4cw\" (UniqueName: \"kubernetes.io/projected/56b360ba-98b8-4d60-b833-9a2d3f842a0d-kube-api-access-nr4cw\") pod \"downloads-76f99584f4-ck5md\" (UID: \"56b360ba-98b8-4d60-b833-9a2d3f842a0d\") " pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:07:44.305557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.305523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:07:44.591810 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.591778 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ck5md"] Sep 16 19:07:44.595057 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:44.595024 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b360ba_98b8_4d60_b833_9a2d3f842a0d.slice/crio-c1b03cdd491d3c4de67c8d260e034811c189be129ffe8b4f5dce9d8a522fc861 WatchSource:0}: Error finding container c1b03cdd491d3c4de67c8d260e034811c189be129ffe8b4f5dce9d8a522fc861: Status 404 returned error can't find the container with id c1b03cdd491d3c4de67c8d260e034811c189be129ffe8b4f5dce9d8a522fc861 Sep 16 19:07:44.818500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.818466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ck5md" event={"ID":"56b360ba-98b8-4d60-b833-9a2d3f842a0d","Type":"ContainerStarted","Data":"c1b03cdd491d3c4de67c8d260e034811c189be129ffe8b4f5dce9d8a522fc861"} Sep 16 19:07:44.818703 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.818518 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676654c77c-4559p" Sep 16 19:07:44.840403 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.840372 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:07:44.844196 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:44.844140 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-676654c77c-4559p"] Sep 16 19:07:46.253258 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:46.253221 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566a56a7-ea6b-46a2-8da1-01325ba9c9af" path="/var/lib/kubelet/pods/566a56a7-ea6b-46a2-8da1-01325ba9c9af/volumes" Sep 16 19:07:53.587735 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.587693 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:07:53.596116 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.596089 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.601747 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.601723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 19:07:53.601896 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.601735 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 19:07:53.601963 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.601772 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 19:07:53.602826 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.602803 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 19:07:53.602951 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.602892 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 19:07:53.603019 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.602987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-67cnv\"" Sep 16 19:07:53.608822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.608796 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:07:53.628630 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:07:53.628596 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-969r4" podUID="8452df11-18f3-46c0-9353-920103de5013" Sep 16 19:07:53.654761 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.654922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.654922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4zh\" (UniqueName: \"kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.654922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.654922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.655121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.654985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756051 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4zh\" (UniqueName: \"kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.756955 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.757005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.756967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.759124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.759100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.759227 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.759161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.775869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.775843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4zh\" (UniqueName: \"kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh\") pod \"console-666996c6c7-ld6vw\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:53.858679 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.858600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-969r4" Sep 16 19:07:53.909153 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:53.909116 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:07:54.194124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:54.193980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:07:54.196494 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:07:54.196463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ae3fde_8f81_46f8_ab98_1f1bf225f062.slice/crio-4b0b223c2ae2e99786900ddd1471396d07f7da986157ad9a164d3a9f50afe7d1 WatchSource:0}: Error finding container 4b0b223c2ae2e99786900ddd1471396d07f7da986157ad9a164d3a9f50afe7d1: Status 404 returned error can't find the container with id 4b0b223c2ae2e99786900ddd1471396d07f7da986157ad9a164d3a9f50afe7d1 Sep 16 19:07:54.864963 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:54.864927 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666996c6c7-ld6vw" event={"ID":"94ae3fde-8f81-46f8-ab98-1f1bf225f062","Type":"ContainerStarted","Data":"4b0b223c2ae2e99786900ddd1471396d07f7da986157ad9a164d3a9f50afe7d1"} Sep 16 19:07:59.203651 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:59.203613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:07:59.206232 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:59.206203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8452df11-18f3-46c0-9353-920103de5013-metrics-tls\") pod \"dns-default-969r4\" (UID: \"8452df11-18f3-46c0-9353-920103de5013\") " pod="openshift-dns/dns-default-969r4" Sep 16 19:07:59.262137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:59.262092 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pvzq\"" Sep 16 19:07:59.270455 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:07:59.270423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-969r4" Sep 16 19:08:01.312238 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.312202 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:08:01.316044 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.316020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.325939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.325917 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 19:08:01.345821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.345776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:08:01.423027 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.422989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423027 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423072 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.423250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.423226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2ns\" (UniqueName: \"kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524660 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.524858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.525098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.524866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2ns\" (UniqueName: \"kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.525834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.525676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.525834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.525740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.525834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.525792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.526052 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.525904 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.527749 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.527718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.533271 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.533244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.537245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.537221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2ns\" (UniqueName: \"kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns\") pod \"console-66848fd7bc-s5njf\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.627385 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.627347 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:01.851851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.851733 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-969r4"] Sep 16 19:08:01.856940 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:08:01.856907 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8452df11_18f3_46c0_9353_920103de5013.slice/crio-281e0c413a6ac51b7fd5c8989b205bdbd61947e830e43619aaa1b2a4516e54a6 WatchSource:0}: Error finding container 281e0c413a6ac51b7fd5c8989b205bdbd61947e830e43619aaa1b2a4516e54a6: Status 404 returned error can't find the container with id 281e0c413a6ac51b7fd5c8989b205bdbd61947e830e43619aaa1b2a4516e54a6 Sep 16 19:08:01.899074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.899034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-969r4" event={"ID":"8452df11-18f3-46c0-9353-920103de5013","Type":"ContainerStarted","Data":"281e0c413a6ac51b7fd5c8989b205bdbd61947e830e43619aaa1b2a4516e54a6"} Sep 16 19:08:01.902571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.902504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ck5md" event={"ID":"56b360ba-98b8-4d60-b833-9a2d3f842a0d","Type":"ContainerStarted","Data":"fea1f1aeb9a329e0dd9261f344bf6cd7c4fa5dac4a460ee6abedd291c7fa8566"} Sep 16 19:08:01.902898 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.902837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:08:01.911889 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.911829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-ck5md" Sep 16 19:08:01.923486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.923463 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:08:01.925737 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:08:01.925701 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c87c252_f12d_4d74_b86f_27ada8ecf4c9.slice/crio-3567e8c8df1f321d9034dc02b830733d068503f46a613a53e3a039d5c28b38b7 WatchSource:0}: Error finding container 3567e8c8df1f321d9034dc02b830733d068503f46a613a53e3a039d5c28b38b7: Status 404 returned error can't find the container with id 3567e8c8df1f321d9034dc02b830733d068503f46a613a53e3a039d5c28b38b7 Sep 16 19:08:01.930561 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:01.930443 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-ck5md" podStartSLOduration=1.8463720879999999 podStartE2EDuration="18.930424628s" totalImagesPullingTime="17.08405254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:01.930276109 +0000 UTC m=+166.136231862" watchObservedRunningTime="2026-09-16 19:08:01.930424628 +0000 UTC m=+166.136380384" Sep 16 19:08:02.914708 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:02.914378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66848fd7bc-s5njf" event={"ID":"1c87c252-f12d-4d74-b86f-27ada8ecf4c9","Type":"ContainerStarted","Data":"3567e8c8df1f321d9034dc02b830733d068503f46a613a53e3a039d5c28b38b7"} Sep 16 19:08:05.932911 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:05.932777 2565 generic.go:356] "Generic (PLEG): container finished" podID="aed24638-847e-4542-82ad-ac5eee3df193" containerID="2f17ac9671a365a1a0fcd1d796b166846a0ba46189fb6cb7d7bb2c25ea9549e9" exitCode=0 Sep 16 19:08:05.932911 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:05.932840 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-5lz2w" event={"ID":"aed24638-847e-4542-82ad-ac5eee3df193","Type":"ContainerDied","Data":"2f17ac9671a365a1a0fcd1d796b166846a0ba46189fb6cb7d7bb2c25ea9549e9"} Sep 16 19:08:05.933468 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:05.933243 2565 scope.go:117] "RemoveContainer" containerID="2f17ac9671a365a1a0fcd1d796b166846a0ba46189fb6cb7d7bb2c25ea9549e9" Sep 16 19:08:06.940859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.940820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-969r4" event={"ID":"8452df11-18f3-46c0-9353-920103de5013","Type":"ContainerStarted","Data":"ad6e564e5b47c56679903b3b599ad6f0b6429c399bb934eaf096bc63d6a3e144"} Sep 16 19:08:06.948430 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.945953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66848fd7bc-s5njf" event={"ID":"1c87c252-f12d-4d74-b86f-27ada8ecf4c9","Type":"ContainerStarted","Data":"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46"} Sep 16 19:08:06.955323 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.953748 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666996c6c7-ld6vw" event={"ID":"94ae3fde-8f81-46f8-ab98-1f1bf225f062","Type":"ContainerStarted","Data":"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0"} Sep 16 19:08:06.957995 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.957912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-5lz2w" event={"ID":"aed24638-847e-4542-82ad-ac5eee3df193","Type":"ContainerStarted","Data":"1986cbc901f88768cef84a09de8625fe2be6c0ecb7a98c07779a2a22948cd6c0"} Sep 16 19:08:06.968404 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.968019 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66848fd7bc-s5njf" podStartSLOduration=1.240446492 podStartE2EDuration="5.967997761s" totalImagesPullingTime="4.727551269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:06.966847695 +0000 UTC m=+171.172803461" watchObservedRunningTime="2026-09-16 19:08:06.967997761 +0000 UTC m=+171.173953514" Sep 16 19:08:06.986656 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:06.986610 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-666996c6c7-ld6vw" podStartSLOduration=1.534566233 podStartE2EDuration="13.98659152s" totalImagesPullingTime="12.452025287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:06.985837787 +0000 UTC m=+171.191793543" watchObservedRunningTime="2026-09-16 19:08:06.98659152 +0000 UTC m=+171.192547277" Sep 16 19:08:07.965360 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:07.965318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-969r4" event={"ID":"8452df11-18f3-46c0-9353-920103de5013","Type":"ContainerStarted","Data":"e88d395de11d6b0bf3e185d68cca51951e71a014998995cc84b01309ba45f660"} Sep 16 19:08:08.970028 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:08.969974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-969r4" Sep 16 19:08:11.628205 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:11.628165 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:11.628681 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:11.628224 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:11.633214 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:11.633190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:11.658558 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:11.658513 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-969r4" podStartSLOduration=136.867350849 podStartE2EDuration="2m21.658497074s" totalImagesPullingTime="4.791146225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:07.992698641 +0000 UTC m=+172.198654391" watchObservedRunningTime="2026-09-16 19:08:11.658497074 +0000 UTC m=+175.864452827" Sep 16 19:08:11.986119 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:11.986091 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:08:12.056590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:12.056557 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:08:13.910084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:13.910050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:08:18.977435 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:18.977400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-969r4" Sep 16 19:08:37.078252 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.078213 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-666996c6c7-ld6vw" podUID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" containerName="console" containerID="cri-o://8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0" gracePeriod=15 Sep 16 19:08:37.345997 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.345919 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:08:37.426127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426095 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426139 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426251 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g4zh\" (UniqueName: \"kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426333 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca\") pod \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\" (UID: \"94ae3fde-8f81-46f8-ab98-1f1bf225f062\") " Sep 16 19:08:37.426619 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426588 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config" (OuterVolumeSpecName: "console-config") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:37.426678 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426647 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:37.426798 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.426773 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca" (OuterVolumeSpecName: "service-ca") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:37.428753 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.428724 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:37.428849 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.428776 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:37.428849 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.428802 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh" (OuterVolumeSpecName: "kube-api-access-5g4zh") pod "94ae3fde-8f81-46f8-ab98-1f1bf225f062" (UID: "94ae3fde-8f81-46f8-ab98-1f1bf225f062"). InnerVolumeSpecName "kube-api-access-5g4zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:37.527610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527575 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-oauth-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:37.527610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527603 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-oauth-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:37.527610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527612 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4zh\" (UniqueName: \"kubernetes.io/projected/94ae3fde-8f81-46f8-ab98-1f1bf225f062-kube-api-access-5g4zh\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:37.527825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527622 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:37.527825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527630 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:37.527825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:37.527639 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94ae3fde-8f81-46f8-ab98-1f1bf225f062-console-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:08:38.088086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.088051 2565 generic.go:356] "Generic (PLEG): container finished" podID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" containerID="8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0" exitCode=0 Sep 16 19:08:38.088500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.088099 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666996c6c7-ld6vw" Sep 16 19:08:38.088500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.088099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666996c6c7-ld6vw" event={"ID":"94ae3fde-8f81-46f8-ab98-1f1bf225f062","Type":"ContainerDied","Data":"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0"} Sep 16 19:08:38.088500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.088254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666996c6c7-ld6vw" event={"ID":"94ae3fde-8f81-46f8-ab98-1f1bf225f062","Type":"ContainerDied","Data":"4b0b223c2ae2e99786900ddd1471396d07f7da986157ad9a164d3a9f50afe7d1"} Sep 16 19:08:38.088500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.088316 2565 scope.go:117] "RemoveContainer" containerID="8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0" Sep 16 19:08:38.097115 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.097100 2565 scope.go:117] "RemoveContainer" containerID="8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0" Sep 16 19:08:38.097400 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:08:38.097380 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0\": container with ID starting with 8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0 not found: ID does not exist" containerID="8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0" Sep 16 19:08:38.097461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.097406 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0"} err="failed to get container status \"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0\": rpc error: code = NotFound desc = could not find container \"8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0\": container with ID starting with 8e49356dbcf8e98d78dd4244a81c13dc801cf70ef9ab6f8c9610d17c51a9f2f0 not found: ID does not exist" Sep 16 19:08:38.110697 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.110674 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:08:38.117414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.117390 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666996c6c7-ld6vw"] Sep 16 19:08:38.252112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:38.252064 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" path="/var/lib/kubelet/pods/94ae3fde-8f81-46f8-ab98-1f1bf225f062/volumes" Sep 16 19:08:52.452321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.452263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-548845546f-2tvlf"] Sep 16 19:08:52.452791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.452575 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" containerName="console" Sep 16 19:08:52.452791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.452585 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" containerName="console" Sep 16 19:08:52.452791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.452654 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ae3fde-8f81-46f8-ab98-1f1bf225f062" containerName="console" Sep 16 19:08:52.455543 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.455524 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.460016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.459997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-86rq8\"" Sep 16 19:08:52.460417 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.460382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 19:08:52.460417 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.460387 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 19:08:52.460649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.460440 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 19:08:52.460649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.460458 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 19:08:52.461376 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.461356 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 19:08:52.469784 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.469761 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 19:08:52.485734 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.485707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-548845546f-2tvlf"] Sep 16 19:08:52.550846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.550806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-serving-certs-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.550846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.550851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.550914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-federate-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.550954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.550987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.551005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgprj\" (UniqueName: \"kubernetes.io/projected/a968364d-0f11-4903-8d76-2e8af58920ae-kube-api-access-mgprj\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.551025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.551257 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.551091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-metrics-client-ca\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.651903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.651859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.651915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgprj\" (UniqueName: \"kubernetes.io/projected/a968364d-0f11-4903-8d76-2e8af58920ae-kube-api-access-mgprj\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.651938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.651985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-metrics-client-ca\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-serving-certs-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-federate-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652552 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-metrics-client-ca\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.652855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.652836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-serving-certs-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.653195 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.653175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-trusted-ca-bundle\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.654707 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.654682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.654938 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.654911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.655242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.655224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-federate-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.655454 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.655436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a968364d-0f11-4903-8d76-2e8af58920ae-telemeter-client-tls\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.666438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.666408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgprj\" (UniqueName: \"kubernetes.io/projected/a968364d-0f11-4903-8d76-2e8af58920ae-kube-api-access-mgprj\") pod \"telemeter-client-548845546f-2tvlf\" (UID: \"a968364d-0f11-4903-8d76-2e8af58920ae\") " pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:52.767130 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:52.767094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" Sep 16 19:08:53.069384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:53.069307 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-548845546f-2tvlf"] Sep 16 19:08:53.073083 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:08:53.073057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda968364d_0f11_4903_8d76_2e8af58920ae.slice/crio-4ac62992ec11a20993e2adbbcb6670c0a583a3b85fda9b6ba4048a5ca9a19db1 WatchSource:0}: Error finding container 4ac62992ec11a20993e2adbbcb6670c0a583a3b85fda9b6ba4048a5ca9a19db1: Status 404 returned error can't find the container with id 4ac62992ec11a20993e2adbbcb6670c0a583a3b85fda9b6ba4048a5ca9a19db1 Sep 16 19:08:53.147850 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:53.147810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" event={"ID":"a968364d-0f11-4903-8d76-2e8af58920ae","Type":"ContainerStarted","Data":"4ac62992ec11a20993e2adbbcb6670c0a583a3b85fda9b6ba4048a5ca9a19db1"} Sep 16 19:08:55.158161 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:55.158073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" event={"ID":"a968364d-0f11-4903-8d76-2e8af58920ae","Type":"ContainerStarted","Data":"201c47d6e3cbfb326cfb1ec2bdb6eba722606defbb2a387043afcbed4e62ae5a"} Sep 16 19:08:56.163958 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.163913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" event={"ID":"a968364d-0f11-4903-8d76-2e8af58920ae","Type":"ContainerStarted","Data":"40b4466561897ab6ea7f1839293115257084b2a20c4a5edc91e913995595fde5"} Sep 16 19:08:56.164371 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.164021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" event={"ID":"a968364d-0f11-4903-8d76-2e8af58920ae","Type":"ContainerStarted","Data":"00c21defba6bd25156bde347e957a97a5fd31f7417563afa64c821ac2b4dc107"} Sep 16 19:08:56.194771 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.194711 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-548845546f-2tvlf" podStartSLOduration=1.320290514 podStartE2EDuration="4.194693847s" totalImagesPullingTime="2.874403333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:56.193673247 +0000 UTC m=+220.399629023" watchObservedRunningTime="2026-09-16 19:08:56.194693847 +0000 UTC m=+220.400649600" Sep 16 19:08:56.299991 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.299912 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:08:56.302154 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.302134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.330879 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.330853 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:08:56.385393 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385742 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.385742 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.385627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggcq2\" (UniqueName: \"kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggcq2\" (UniqueName: \"kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.486920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.487197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.486923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.487568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.487537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.487703 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.487682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.487767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.487704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.487804 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.487776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.489957 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.489913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.490448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.490421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.500377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.500342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggcq2\" (UniqueName: \"kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2\") pod \"console-797549dc98-chdgk\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.611360 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.611256 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:08:56.896029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:56.896005 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:08:56.898110 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:08:56.898079 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c6b1c1_6695_48cf_8891_83c40c45d5c7.slice/crio-30af5412713811a3c11840e9c1abbe2d68432f7696f78e98871cb35bb389d2cb WatchSource:0}: Error finding container 30af5412713811a3c11840e9c1abbe2d68432f7696f78e98871cb35bb389d2cb: Status 404 returned error can't find the container with id 30af5412713811a3c11840e9c1abbe2d68432f7696f78e98871cb35bb389d2cb Sep 16 19:08:57.169334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:57.169228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797549dc98-chdgk" event={"ID":"02c6b1c1-6695-48cf-8891-83c40c45d5c7","Type":"ContainerStarted","Data":"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b"} Sep 16 19:08:57.169334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:08:57.169303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797549dc98-chdgk" event={"ID":"02c6b1c1-6695-48cf-8891-83c40c45d5c7","Type":"ContainerStarted","Data":"30af5412713811a3c11840e9c1abbe2d68432f7696f78e98871cb35bb389d2cb"} Sep 16 19:09:06.612300 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:06.612231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:09:06.612713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:06.612332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:09:06.617186 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:06.617162 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:09:06.643621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:06.643567 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-797549dc98-chdgk" podStartSLOduration=10.643551593 podStartE2EDuration="10.643551593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:57.196912891 +0000 UTC m=+221.402868656" watchObservedRunningTime="2026-09-16 19:09:06.643551593 +0000 UTC m=+230.849507346" Sep 16 19:09:07.212965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:07.212936 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:09:07.305066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:07.305031 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:09:32.324610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.324544 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66848fd7bc-s5njf" podUID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" containerName="console" containerID="cri-o://479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46" gracePeriod=15 Sep 16 19:09:32.561842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.561815 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:09:32.585919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.585833 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.585922 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.585953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.585991 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586152 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g2ns\" (UniqueName: \"kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca\") pod \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\" (UID: \"1c87c252-f12d-4d74-b86f-27ada8ecf4c9\") " Sep 16 19:09:32.586448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586367 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config" (OuterVolumeSpecName: "console-config") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:32.586448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586383 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:32.586448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586388 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:32.586599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586485 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-trusted-ca-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.586599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586497 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-oauth-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.586599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586507 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.586905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.586851 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:32.588179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.588156 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:09:32.588316 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.588277 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:09:32.588370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.588327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns" (OuterVolumeSpecName: "kube-api-access-7g2ns") pod "1c87c252-f12d-4d74-b86f-27ada8ecf4c9" (UID: "1c87c252-f12d-4d74-b86f-27ada8ecf4c9"). InnerVolumeSpecName "kube-api-access-7g2ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:09:32.687389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.687351 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-oauth-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.687389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.687384 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2ns\" (UniqueName: \"kubernetes.io/projected/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-kube-api-access-7g2ns\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.687389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.687393 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:32.687622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:32.687402 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c87c252-f12d-4d74-b86f-27ada8ecf4c9-console-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:09:33.314476 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.314442 2565 generic.go:356] "Generic (PLEG): container finished" podID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" containerID="479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46" exitCode=0 Sep 16 19:09:33.314663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.314490 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66848fd7bc-s5njf" Sep 16 19:09:33.314663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.314537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66848fd7bc-s5njf" event={"ID":"1c87c252-f12d-4d74-b86f-27ada8ecf4c9","Type":"ContainerDied","Data":"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46"} Sep 16 19:09:33.314663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.314590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66848fd7bc-s5njf" event={"ID":"1c87c252-f12d-4d74-b86f-27ada8ecf4c9","Type":"ContainerDied","Data":"3567e8c8df1f321d9034dc02b830733d068503f46a613a53e3a039d5c28b38b7"} Sep 16 19:09:33.314663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.314617 2565 scope.go:117] "RemoveContainer" containerID="479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46" Sep 16 19:09:33.325804 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.325571 2565 scope.go:117] "RemoveContainer" containerID="479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46" Sep 16 19:09:33.326127 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:09:33.326103 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46\": container with ID starting with 479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46 not found: ID does not exist" containerID="479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46" Sep 16 19:09:33.326170 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.326141 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46"} err="failed to get container status \"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46\": rpc error: code = NotFound desc = could not find container \"479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46\": container with ID starting with 479bf6205a0bc7ac0a4d142c5f05cc54a126ccdfe4f83e90cb5768a8311aad46 not found: ID does not exist" Sep 16 19:09:33.338271 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.338246 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:09:33.348471 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:33.348441 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66848fd7bc-s5njf"] Sep 16 19:09:34.252471 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:09:34.252436 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" path="/var/lib/kubelet/pods/1c87c252-f12d-4d74-b86f-27ada8ecf4c9/volumes" Sep 16 19:10:16.176719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:16.176432 2565 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 19:10:42.228906 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.228872 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:10:42.231236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.229167 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" containerName="console" Sep 16 19:10:42.231236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.229179 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" containerName="console" Sep 16 19:10:42.231236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.229244 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c87c252-f12d-4d74-b86f-27ada8ecf4c9" containerName="console" Sep 16 19:10:42.232125 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.232106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.258624 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.258598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:10:42.346530 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hmf\" (UniqueName: \"kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.346698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.346699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447512 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hmf\" (UniqueName: \"kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.447982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.447706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.448388 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.448359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.448719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.448395 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.448719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.448437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.448840 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.448437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.450224 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.450203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.450406 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.450387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.457816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.457794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hmf\" (UniqueName: \"kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf\") pod \"console-7f57d597bc-8rptz\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.541730 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.541653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:42.819954 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:42.819795 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:10:42.822591 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:10:42.822559 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7760cba9_3055_4c66_a344_478182cfe61c.slice/crio-50c0aa5f19a378a1449d2941556e169e501f8270a3767fa85b921c24b0b167d0 WatchSource:0}: Error finding container 50c0aa5f19a378a1449d2941556e169e501f8270a3767fa85b921c24b0b167d0: Status 404 returned error can't find the container with id 50c0aa5f19a378a1449d2941556e169e501f8270a3767fa85b921c24b0b167d0 Sep 16 19:10:43.587370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:43.587331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f57d597bc-8rptz" event={"ID":"7760cba9-3055-4c66-a344-478182cfe61c","Type":"ContainerStarted","Data":"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177"} Sep 16 19:10:43.587869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:43.587388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f57d597bc-8rptz" event={"ID":"7760cba9-3055-4c66-a344-478182cfe61c","Type":"ContainerStarted","Data":"50c0aa5f19a378a1449d2941556e169e501f8270a3767fa85b921c24b0b167d0"} Sep 16 19:10:43.619613 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:43.619561 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f57d597bc-8rptz" podStartSLOduration=1.619546486 podStartE2EDuration="1.619546486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:43.619034215 +0000 UTC m=+327.824989968" watchObservedRunningTime="2026-09-16 19:10:43.619546486 +0000 UTC m=+327.825502238" Sep 16 19:10:52.541939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:52.541901 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:52.542471 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:52.541967 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:52.546792 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:52.546769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:52.629458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:52.629430 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:10:52.723540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:10:52.723510 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:11:07.758227 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.758146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24"] Sep 16 19:11:07.761565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.761549 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.765018 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.764994 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 19:11:07.765147 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.765050 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b6684\"" Sep 16 19:11:07.767969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.767943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 19:11:07.807242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.807213 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24"] Sep 16 19:11:07.865829 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.865795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbtz\" (UniqueName: \"kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.865981 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.865849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.865981 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.865917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.967112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.967072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbtz\" (UniqueName: \"kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.967310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.967123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.967310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.967155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.967534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.967518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.967570 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.967549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:07.995673 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:07.995639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbtz\" (UniqueName: \"kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:08.071296 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:08.071190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:08.360611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:08.360547 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24"] Sep 16 19:11:08.363683 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:11:08.363659 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf534000_4d19_40b2_916e_5f173c0a56ba.slice/crio-c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d WatchSource:0}: Error finding container c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d: Status 404 returned error can't find the container with id c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d Sep 16 19:11:08.365722 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:08.365706 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:11:08.689252 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:08.689174 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerStarted","Data":"c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d"} Sep 16 19:11:14.718710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:14.718614 2565 generic.go:356] "Generic (PLEG): container finished" podID="af534000-4d19-40b2-916e-5f173c0a56ba" containerID="43170611d65b60f9ddf0d813721add17a0d154f705bdbe1da85d2255b0da25dd" exitCode=0 Sep 16 19:11:14.718710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:14.718673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerDied","Data":"43170611d65b60f9ddf0d813721add17a0d154f705bdbe1da85d2255b0da25dd"} Sep 16 19:11:16.728461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:16.728432 2565 generic.go:356] "Generic (PLEG): container finished" podID="af534000-4d19-40b2-916e-5f173c0a56ba" containerID="ffd688a0326762a024355eafac78ef9dc28003486649a5b82fdf15b8a676813d" exitCode=0 Sep 16 19:11:16.728774 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:16.728525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerDied","Data":"ffd688a0326762a024355eafac78ef9dc28003486649a5b82fdf15b8a676813d"} Sep 16 19:11:17.743887 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:17.743849 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-797549dc98-chdgk" podUID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" containerName="console" containerID="cri-o://8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b" gracePeriod=15 Sep 16 19:11:18.011889 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.011862 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:11:18.161550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161555 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggcq2\" (UniqueName: \"kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161636 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161677 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161701 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.161796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.161725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert\") pod \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\" (UID: \"02c6b1c1-6695-48cf-8891-83c40c45d5c7\") " Sep 16 19:11:18.162031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.162003 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config" (OuterVolumeSpecName: "console-config") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:18.162192 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.162160 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:18.162343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.162321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:18.162343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.162327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:18.164014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.163989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:11:18.164467 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.164444 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:11:18.164467 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.164448 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2" (OuterVolumeSpecName: "kube-api-access-ggcq2") pod "02c6b1c1-6695-48cf-8891-83c40c45d5c7" (UID: "02c6b1c1-6695-48cf-8891-83c40c45d5c7"). InnerVolumeSpecName "kube-api-access-ggcq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:11:18.262706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262683 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262705 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-trusted-ca-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262714 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262722 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-oauth-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262731 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-oauth-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262738 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02c6b1c1-6695-48cf-8891-83c40c45d5c7-console-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.262841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.262746 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggcq2\" (UniqueName: \"kubernetes.io/projected/02c6b1c1-6695-48cf-8891-83c40c45d5c7-kube-api-access-ggcq2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:18.739937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.739898 2565 generic.go:356] "Generic (PLEG): container finished" podID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" containerID="8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b" exitCode=0 Sep 16 19:11:18.739937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.739944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797549dc98-chdgk" event={"ID":"02c6b1c1-6695-48cf-8891-83c40c45d5c7","Type":"ContainerDied","Data":"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b"} Sep 16 19:11:18.740182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.739957 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797549dc98-chdgk" Sep 16 19:11:18.740182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.739991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797549dc98-chdgk" event={"ID":"02c6b1c1-6695-48cf-8891-83c40c45d5c7","Type":"ContainerDied","Data":"30af5412713811a3c11840e9c1abbe2d68432f7696f78e98871cb35bb389d2cb"} Sep 16 19:11:18.740182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.740020 2565 scope.go:117] "RemoveContainer" containerID="8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b" Sep 16 19:11:18.749890 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.749683 2565 scope.go:117] "RemoveContainer" containerID="8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b" Sep 16 19:11:18.750145 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:18.749993 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b\": container with ID starting with 8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b not found: ID does not exist" containerID="8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b" Sep 16 19:11:18.750145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.750031 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b"} err="failed to get container status \"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b\": rpc error: code = NotFound desc = could not find container \"8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b\": container with ID starting with 8632995c68456edc02b256a35c7a350c8c9a6b1083559070d524772a0838250b not found: ID does not exist" Sep 16 19:11:18.773223 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.773193 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:11:18.785375 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:18.785349 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-797549dc98-chdgk"] Sep 16 19:11:20.253025 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:20.252988 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" path="/var/lib/kubelet/pods/02c6b1c1-6695-48cf-8891-83c40c45d5c7/volumes" Sep 16 19:11:23.765153 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:23.765127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerStarted","Data":"26a16b1a6acf1f22272696f8f614d59edb432ea50753d3e4e554fa1799d27f0d"} Sep 16 19:11:24.773832 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.773800 2565 generic.go:356] "Generic (PLEG): container finished" podID="af534000-4d19-40b2-916e-5f173c0a56ba" containerID="26a16b1a6acf1f22272696f8f614d59edb432ea50753d3e4e554fa1799d27f0d" exitCode=0 Sep 16 19:11:24.774299 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.773868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerDied","Data":"26a16b1a6acf1f22272696f8f614d59edb432ea50753d3e4e554fa1799d27f0d"} Sep 16 19:11:24.895173 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.895148 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:24.920009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.919984 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle\") pod \"af534000-4d19-40b2-916e-5f173c0a56ba\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " Sep 16 19:11:24.920210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.920039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbbtz\" (UniqueName: \"kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz\") pod \"af534000-4d19-40b2-916e-5f173c0a56ba\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " Sep 16 19:11:24.920210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.920103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util\") pod \"af534000-4d19-40b2-916e-5f173c0a56ba\" (UID: \"af534000-4d19-40b2-916e-5f173c0a56ba\") " Sep 16 19:11:24.920883 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.920852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle" (OuterVolumeSpecName: "bundle") pod "af534000-4d19-40b2-916e-5f173c0a56ba" (UID: "af534000-4d19-40b2-916e-5f173c0a56ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:11:24.922722 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.922700 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz" (OuterVolumeSpecName: "kube-api-access-kbbtz") pod "af534000-4d19-40b2-916e-5f173c0a56ba" (UID: "af534000-4d19-40b2-916e-5f173c0a56ba"). InnerVolumeSpecName "kube-api-access-kbbtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:11:24.924656 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:24.924636 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util" (OuterVolumeSpecName: "util") pod "af534000-4d19-40b2-916e-5f173c0a56ba" (UID: "af534000-4d19-40b2-916e-5f173c0a56ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:11:25.020865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.020829 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:25.020865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.020859 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbtz\" (UniqueName: \"kubernetes.io/projected/af534000-4d19-40b2-916e-5f173c0a56ba-kube-api-access-kbbtz\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:25.020865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.020871 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af534000-4d19-40b2-916e-5f173c0a56ba-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:11:25.780431 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.780394 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" event={"ID":"af534000-4d19-40b2-916e-5f173c0a56ba","Type":"ContainerDied","Data":"c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d"} Sep 16 19:11:25.780431 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.780427 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mh24" Sep 16 19:11:25.780883 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:25.780449 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c01c46174be7a1b64757f22009fa4011a10ef8281f449597c2f0f00dc3afb94d" Sep 16 19:11:32.972048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972015 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6"] Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972381 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="pull" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972395 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="pull" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972406 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="util" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972411 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="util" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972421 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="extract" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972429 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="extract" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972439 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" containerName="console" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972444 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" containerName="console" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972492 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af534000-4d19-40b2-916e-5f173c0a56ba" containerName="extract" Sep 16 19:11:32.972568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:32.972502 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c6b1c1-6695-48cf-8891-83c40c45d5c7" containerName="console" Sep 16 19:11:33.008876 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.008842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.016235 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.016206 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 19:11:33.016620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.016601 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-h7p4l\"" Sep 16 19:11:33.016718 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.016659 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 19:11:33.017052 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.017029 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 19:11:33.018389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.018367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6"] Sep 16 19:11:33.079958 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.079928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.080110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.079986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrpd\" (UniqueName: \"kubernetes.io/projected/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-kube-api-access-cmrpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.180746 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.180710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrpd\" (UniqueName: \"kubernetes.io/projected/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-kube-api-access-cmrpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.180918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.180776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.183393 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.183363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.194091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.194067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrpd\" (UniqueName: \"kubernetes.io/projected/f03185ff-1521-46a1-b73a-8eb8eaa5ed75-kube-api-access-cmrpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tp7v6\" (UID: \"f03185ff-1521-46a1-b73a-8eb8eaa5ed75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.318844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.318752 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:33.652245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.652221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6"] Sep 16 19:11:33.655220 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:11:33.655191 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03185ff_1521_46a1_b73a_8eb8eaa5ed75.slice/crio-44d116437d925b91a96aea9c9bc24fdccafa7a9649844b58abdba79930ddfc8a WatchSource:0}: Error finding container 44d116437d925b91a96aea9c9bc24fdccafa7a9649844b58abdba79930ddfc8a: Status 404 returned error can't find the container with id 44d116437d925b91a96aea9c9bc24fdccafa7a9649844b58abdba79930ddfc8a Sep 16 19:11:33.812993 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:33.812961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" event={"ID":"f03185ff-1521-46a1-b73a-8eb8eaa5ed75","Type":"ContainerStarted","Data":"44d116437d925b91a96aea9c9bc24fdccafa7a9649844b58abdba79930ddfc8a"} Sep 16 19:11:36.829860 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:36.829816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" event={"ID":"f03185ff-1521-46a1-b73a-8eb8eaa5ed75","Type":"ContainerStarted","Data":"2d11523adc2cf8552bede0db1a34f0ce5a0d39fb1e601dad866c4997fb6c0459"} Sep 16 19:11:36.830301 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:36.829946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:11:38.108898 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.108830 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" podStartSLOduration=3.171325001 podStartE2EDuration="6.108808038s" totalImagesPullingTime="2.937483037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:36.867177934 +0000 UTC m=+381.073133688" watchObservedRunningTime="2026-09-16 19:11:38.108808038 +0000 UTC m=+382.314763793" Sep 16 19:11:38.110361 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.110337 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hrnhx"] Sep 16 19:11:38.112731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.112709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.115201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.115176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.115322 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.115216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.115374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.115340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vv8\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-kube-api-access-d7vv8\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.115591 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.115574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-js65r\"" Sep 16 19:11:38.116111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.116094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 19:11:38.118661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.118645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 16 19:11:38.135463 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.135435 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hrnhx"] Sep 16 19:11:38.215934 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.215901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.215942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.215984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vv8\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-kube-api-access-d7vv8\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216077 2565 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216092 2565 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: configmap "keda-ocp-cabundle" not found Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216103 2565 secret.go:281] references non-existent secret key: ca.crt Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216115 2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 19:11:38.216126 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216131 2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hrnhx: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 19:11:38.216491 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216153 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0 podName:ff17a408-e27e-4a06-9793-1089459e1d87 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:38.716135779 +0000 UTC m=+382.922091509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0") pod "keda-operator-579c8958f7-hrnhx" (UID: "ff17a408-e27e-4a06-9793-1089459e1d87") : configmap "keda-ocp-cabundle" not found Sep 16 19:11:38.216491 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:11:38.216188 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates podName:ff17a408-e27e-4a06-9793-1089459e1d87 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:38.716172488 +0000 UTC m=+382.922128236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates") pod "keda-operator-579c8958f7-hrnhx" (UID: "ff17a408-e27e-4a06-9793-1089459e1d87") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 19:11:38.231332 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.231309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vv8\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-kube-api-access-d7vv8\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.719206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.719166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.719206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.719205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.719813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.719791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff17a408-e27e-4a06-9793-1089459e1d87-cabundle0\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.721725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.721701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff17a408-e27e-4a06-9793-1089459e1d87-certificates\") pod \"keda-operator-579c8958f7-hrnhx\" (UID: \"ff17a408-e27e-4a06-9793-1089459e1d87\") " pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.723224 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.723207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:38.870624 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.870590 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krw69"] Sep 16 19:11:38.873098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.873080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:38.879199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.879178 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 16 19:11:38.893535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.893507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krw69"] Sep 16 19:11:38.920772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.920734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kckz\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-kube-api-access-9kckz\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:38.920916 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.920780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ad8f2a97-27f4-48ca-acd8-18942598885b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:38.920916 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:38.920813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.004257 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.004198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hrnhx"] Sep 16 19:11:39.006960 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:11:39.006933 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff17a408_e27e_4a06_9793_1089459e1d87.slice/crio-e7ad5bc5d855125fb9430fdbb99adbc9d01e20561f77a141393db51334613bee WatchSource:0}: Error finding container e7ad5bc5d855125fb9430fdbb99adbc9d01e20561f77a141393db51334613bee: Status 404 returned error can't find the container with id e7ad5bc5d855125fb9430fdbb99adbc9d01e20561f77a141393db51334613bee Sep 16 19:11:39.022189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.022163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kckz\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-kube-api-access-9kckz\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.022327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.022212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ad8f2a97-27f4-48ca-acd8-18942598885b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.022327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.022261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.022618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.022599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ad8f2a97-27f4-48ca-acd8-18942598885b-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.025222 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.025199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-certificates\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.041475 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.041449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kckz\" (UniqueName: \"kubernetes.io/projected/ad8f2a97-27f4-48ca-acd8-18942598885b-kube-api-access-9kckz\") pod \"keda-metrics-apiserver-77857b7b98-krw69\" (UID: \"ad8f2a97-27f4-48ca-acd8-18942598885b\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.155727 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.155662 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79ktq"] Sep 16 19:11:39.158182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.158160 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.161295 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.161261 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 16 19:11:39.176152 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.176128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79ktq"] Sep 16 19:11:39.183449 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.183426 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:39.224611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.224576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-certificates\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.224773 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.224627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scxw\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-kube-api-access-2scxw\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.326319 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.326202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-certificates\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.326319 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.326257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scxw\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-kube-api-access-2scxw\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.328936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.328909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-certificates\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.340084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.340051 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scxw\" (UniqueName: \"kubernetes.io/projected/3ea8b5bb-8305-409a-842a-4cf4d22f2f88-kube-api-access-2scxw\") pod \"keda-admission-6c4d864b86-79ktq\" (UID: \"3ea8b5bb-8305-409a-842a-4cf4d22f2f88\") " pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.468101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.468066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:39.476763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.476742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-krw69"] Sep 16 19:11:39.479301 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:11:39.479254 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8f2a97_27f4_48ca_acd8_18942598885b.slice/crio-20ca2c6671eb956a2a747c814c375f2c80053cc8010b64e0e4104274add9c058 WatchSource:0}: Error finding container 20ca2c6671eb956a2a747c814c375f2c80053cc8010b64e0e4104274add9c058: Status 404 returned error can't find the container with id 20ca2c6671eb956a2a747c814c375f2c80053cc8010b64e0e4104274add9c058 Sep 16 19:11:39.750807 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.750779 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-79ktq"] Sep 16 19:11:39.752919 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:11:39.752890 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea8b5bb_8305_409a_842a_4cf4d22f2f88.slice/crio-ba93ce3b4f64c3ccc91da69bb2703bb0e4f36fb00044055ed04d110dcfc18c61 WatchSource:0}: Error finding container ba93ce3b4f64c3ccc91da69bb2703bb0e4f36fb00044055ed04d110dcfc18c61: Status 404 returned error can't find the container with id ba93ce3b4f64c3ccc91da69bb2703bb0e4f36fb00044055ed04d110dcfc18c61 Sep 16 19:11:39.845358 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.845310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" event={"ID":"ad8f2a97-27f4-48ca-acd8-18942598885b","Type":"ContainerStarted","Data":"20ca2c6671eb956a2a747c814c375f2c80053cc8010b64e0e4104274add9c058"} Sep 16 19:11:39.846491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.846464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" event={"ID":"3ea8b5bb-8305-409a-842a-4cf4d22f2f88","Type":"ContainerStarted","Data":"ba93ce3b4f64c3ccc91da69bb2703bb0e4f36fb00044055ed04d110dcfc18c61"} Sep 16 19:11:39.847520 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:39.847501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" event={"ID":"ff17a408-e27e-4a06-9793-1089459e1d87","Type":"ContainerStarted","Data":"e7ad5bc5d855125fb9430fdbb99adbc9d01e20561f77a141393db51334613bee"} Sep 16 19:11:41.861346 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:41.861311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" event={"ID":"3ea8b5bb-8305-409a-842a-4cf4d22f2f88","Type":"ContainerStarted","Data":"7f77e0c3a1f87afd6678fb2f61a3904479a437c49666cbb022b5e944a874e90d"} Sep 16 19:11:41.861794 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:41.861455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:11:41.889936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:41.889758 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" podStartSLOduration=1.286537872 podStartE2EDuration="2.889736268s" totalImagesPullingTime="1.603198396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:41.887929139 +0000 UTC m=+386.093884892" watchObservedRunningTime="2026-09-16 19:11:41.889736268 +0000 UTC m=+386.095692033" Sep 16 19:11:43.872450 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.872347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" event={"ID":"ad8f2a97-27f4-48ca-acd8-18942598885b","Type":"ContainerStarted","Data":"1f109bd3fa4ebe5bcc4089e78b89d9a853a3dcbf9c0d2c442d38d517db7d0b27"} Sep 16 19:11:43.872450 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.872416 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:43.873953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.873930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" event={"ID":"ff17a408-e27e-4a06-9793-1089459e1d87","Type":"ContainerStarted","Data":"c325448a4ab753ca99a8ea3925b652a47ce8647e081004fecc3642d5b70334f2"} Sep 16 19:11:43.874074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.873977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:11:43.929991 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.929937 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" podStartSLOduration=1.937094918 podStartE2EDuration="5.929920358s" totalImagesPullingTime="3.99282544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:43.928217442 +0000 UTC m=+388.134173194" watchObservedRunningTime="2026-09-16 19:11:43.929920358 +0000 UTC m=+388.135876112" Sep 16 19:11:43.987087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:43.987032 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" podStartSLOduration=1.517188162 podStartE2EDuration="5.987013854s" totalImagesPullingTime="4.469825692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:43.985673239 +0000 UTC m=+388.191628991" watchObservedRunningTime="2026-09-16 19:11:43.987013854 +0000 UTC m=+388.192969606" Sep 16 19:11:54.883946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:54.883917 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-krw69" Sep 16 19:11:57.837840 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:11:57.837807 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tp7v6" Sep 16 19:12:02.869365 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:02.869332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-79ktq" Sep 16 19:12:04.881597 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:04.881571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-hrnhx" Sep 16 19:12:34.160115 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.160078 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g"] Sep 16 19:12:34.166508 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.166487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.169937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.169918 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 19:12:34.170768 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.170755 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b6684\"" Sep 16 19:12:34.170869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.170780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 19:12:34.193482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.193452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g"] Sep 16 19:12:34.210858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.210830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.211003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.210891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjkr\" (UniqueName: \"kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.211003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.210929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.312030 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.312000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjkr\" (UniqueName: \"kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.312214 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.312038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.312214 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.312089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.312440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.312423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.312490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.312469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.322049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.322030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjkr\" (UniqueName: \"kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.476028 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.475994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:34.754523 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:34.754498 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g"] Sep 16 19:12:34.756722 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:12:34.756694 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e2db6d_f1b0_4b6e_963b_f1d1e192f661.slice/crio-99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da WatchSource:0}: Error finding container 99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da: Status 404 returned error can't find the container with id 99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da Sep 16 19:12:35.118851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:35.118759 2565 generic.go:356] "Generic (PLEG): container finished" podID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerID="0d39b3dba4b9278690d54586807c0f35d9c10b42625aaebc8921bfd0d077c8a9" exitCode=0 Sep 16 19:12:35.119007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:35.118853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" event={"ID":"26e2db6d-f1b0-4b6e-963b-f1d1e192f661","Type":"ContainerDied","Data":"0d39b3dba4b9278690d54586807c0f35d9c10b42625aaebc8921bfd0d077c8a9"} Sep 16 19:12:35.119007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:35.118910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" event={"ID":"26e2db6d-f1b0-4b6e-963b-f1d1e192f661","Type":"ContainerStarted","Data":"99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da"} Sep 16 19:12:38.136632 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:38.136591 2565 generic.go:356] "Generic (PLEG): container finished" podID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerID="483184e81d2de0669321d3c2a0c272498354ac6aa8b42c15ddeee83a24077cb8" exitCode=0 Sep 16 19:12:38.137111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:38.136675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" event={"ID":"26e2db6d-f1b0-4b6e-963b-f1d1e192f661","Type":"ContainerDied","Data":"483184e81d2de0669321d3c2a0c272498354ac6aa8b42c15ddeee83a24077cb8"} Sep 16 19:12:39.143771 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:39.143736 2565 generic.go:356] "Generic (PLEG): container finished" podID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerID="6cbe05f8b6e3d2dc31546c6cce2e956e3bafe52b3973689afe4c386ed37cb3b2" exitCode=0 Sep 16 19:12:39.144143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:39.143816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" event={"ID":"26e2db6d-f1b0-4b6e-963b-f1d1e192f661","Type":"ContainerDied","Data":"6cbe05f8b6e3d2dc31546c6cce2e956e3bafe52b3973689afe4c386ed37cb3b2"} Sep 16 19:12:40.275624 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.275599 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:40.363133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.363096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbjkr\" (UniqueName: \"kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr\") pod \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " Sep 16 19:12:40.363339 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.363141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util\") pod \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " Sep 16 19:12:40.363339 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.363160 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle\") pod \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\" (UID: \"26e2db6d-f1b0-4b6e-963b-f1d1e192f661\") " Sep 16 19:12:40.364024 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.363993 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle" (OuterVolumeSpecName: "bundle") pod "26e2db6d-f1b0-4b6e-963b-f1d1e192f661" (UID: "26e2db6d-f1b0-4b6e-963b-f1d1e192f661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:12:40.365445 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.365424 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr" (OuterVolumeSpecName: "kube-api-access-cbjkr") pod "26e2db6d-f1b0-4b6e-963b-f1d1e192f661" (UID: "26e2db6d-f1b0-4b6e-963b-f1d1e192f661"). InnerVolumeSpecName "kube-api-access-cbjkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:12:40.368873 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.368832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util" (OuterVolumeSpecName: "util") pod "26e2db6d-f1b0-4b6e-963b-f1d1e192f661" (UID: "26e2db6d-f1b0-4b6e-963b-f1d1e192f661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:12:40.463901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.463817 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjkr\" (UniqueName: \"kubernetes.io/projected/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-kube-api-access-cbjkr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:12:40.463901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.463844 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:12:40.463901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:40.463853 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26e2db6d-f1b0-4b6e-963b-f1d1e192f661-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:12:41.154402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:41.154363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" event={"ID":"26e2db6d-f1b0-4b6e-963b-f1d1e192f661","Type":"ContainerDied","Data":"99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da"} Sep 16 19:12:41.154402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:41.154388 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qnj5g" Sep 16 19:12:41.154402 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:41.154409 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99921e75963207c22e1a90ea36f9ae54ca525e3e4467e9423489adf38f9038da" Sep 16 19:12:48.775297 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775249 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s"] Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775603 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="util" Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775615 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="util" Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775624 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="pull" Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775629 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="pull" Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775637 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="extract" Sep 16 19:12:48.775664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775644 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="extract" Sep 16 19:12:48.775847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.775707 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e2db6d-f1b0-4b6e-963b-f1d1e192f661" containerName="extract" Sep 16 19:12:48.779326 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.779309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.784862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.784837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mnj7r\"" Sep 16 19:12:48.785818 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.785801 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 19:12:48.786114 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.786091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:12:48.807891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.807859 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s"] Sep 16 19:12:48.835521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.835488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xj2\" (UniqueName: \"kubernetes.io/projected/2514e933-5253-40cc-b2c7-f606f7e8de40-kube-api-access-f7xj2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.835666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.835620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2514e933-5253-40cc-b2c7-f606f7e8de40-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.936464 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.936426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2514e933-5253-40cc-b2c7-f606f7e8de40-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.936627 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.936485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xj2\" (UniqueName: \"kubernetes.io/projected/2514e933-5253-40cc-b2c7-f606f7e8de40-kube-api-access-f7xj2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.936827 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.936807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2514e933-5253-40cc-b2c7-f606f7e8de40-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:48.951263 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:48.951229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xj2\" (UniqueName: \"kubernetes.io/projected/2514e933-5253-40cc-b2c7-f606f7e8de40-kube-api-access-f7xj2\") pod \"cert-manager-operator-controller-manager-684f79c9b9-clf6s\" (UID: \"2514e933-5253-40cc-b2c7-f606f7e8de40\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:49.088454 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:49.088366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" Sep 16 19:12:49.395570 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:49.395542 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s"] Sep 16 19:12:49.396305 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:12:49.396260 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2514e933_5253_40cc_b2c7_f606f7e8de40.slice/crio-fa90c01ffbddd494b5a25666b24a514545e45a1eb61a0936b217e7fcc1992a4e WatchSource:0}: Error finding container fa90c01ffbddd494b5a25666b24a514545e45a1eb61a0936b217e7fcc1992a4e: Status 404 returned error can't find the container with id fa90c01ffbddd494b5a25666b24a514545e45a1eb61a0936b217e7fcc1992a4e Sep 16 19:12:50.195342 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:50.195304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" event={"ID":"2514e933-5253-40cc-b2c7-f606f7e8de40","Type":"ContainerStarted","Data":"fa90c01ffbddd494b5a25666b24a514545e45a1eb61a0936b217e7fcc1992a4e"} Sep 16 19:12:52.207415 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:52.207370 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" event={"ID":"2514e933-5253-40cc-b2c7-f606f7e8de40","Type":"ContainerStarted","Data":"23555035714ca2418770aba0fa13715a421d2f2310e3b3dc23ebff12d50d5ae5"} Sep 16 19:12:52.296164 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:12:52.296110 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-clf6s" podStartSLOduration=2.428843214 podStartE2EDuration="4.296094937s" totalImagesPullingTime="1.867251723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:52.28507288 +0000 UTC m=+456.491028632" watchObservedRunningTime="2026-09-16 19:12:52.296094937 +0000 UTC m=+456.502050689" Sep 16 19:13:08.795966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.795926 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf"] Sep 16 19:13:08.799381 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.799364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.802601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.802573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b6684\"" Sep 16 19:13:08.802730 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.802618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 19:13:08.803416 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.803402 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 19:13:08.814559 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.814537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.814649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.814583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87n9v\" (UniqueName: \"kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.814649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.814640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.914329 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.914277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf"] Sep 16 19:13:08.915191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.915169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.915293 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.915262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.915356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.915340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87n9v\" (UniqueName: \"kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.915618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.915600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.915691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.915620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:08.934179 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:08.934147 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87n9v\" (UniqueName: \"kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:09.108440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:09.108353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:09.405381 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:09.405355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf"] Sep 16 19:13:09.407928 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:13:09.407892 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e91518d_0cfb_4bd1_bd7b_5421fdf32fd7.slice/crio-86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4 WatchSource:0}: Error finding container 86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4: Status 404 returned error can't find the container with id 86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4 Sep 16 19:13:10.292542 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:10.292509 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerID="e2498d2cdc9ed5d5835ef27b5ee9644d6974ad2d56d3253b9f2b26d86f0a65da" exitCode=0 Sep 16 19:13:10.292954 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:10.292603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" event={"ID":"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7","Type":"ContainerDied","Data":"e2498d2cdc9ed5d5835ef27b5ee9644d6974ad2d56d3253b9f2b26d86f0a65da"} Sep 16 19:13:10.292954 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:10.292667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" event={"ID":"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7","Type":"ContainerStarted","Data":"86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4"} Sep 16 19:13:13.310463 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:13.310425 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerID="86d37dbe0ff31ff36dc0b803235ee420c5d2af4c46f65cea1566c2d7e44dd7ec" exitCode=0 Sep 16 19:13:13.310892 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:13.310498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" event={"ID":"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7","Type":"ContainerDied","Data":"86d37dbe0ff31ff36dc0b803235ee420c5d2af4c46f65cea1566c2d7e44dd7ec"} Sep 16 19:13:14.317531 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:14.317494 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerID="09cd9983ca10702fa5b5212dcb8adef5e9cfb3a56fef0c39677dd0fbf81cf402" exitCode=0 Sep 16 19:13:14.317920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:14.317559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" event={"ID":"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7","Type":"ContainerDied","Data":"09cd9983ca10702fa5b5212dcb8adef5e9cfb3a56fef0c39677dd0fbf81cf402"} Sep 16 19:13:15.450973 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.450946 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:15.469320 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.469277 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle\") pod \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " Sep 16 19:13:15.469432 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.469375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util\") pod \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " Sep 16 19:13:15.469432 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.469402 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87n9v\" (UniqueName: \"kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v\") pod \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\" (UID: \"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7\") " Sep 16 19:13:15.469725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.469700 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle" (OuterVolumeSpecName: "bundle") pod "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" (UID: "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:13:15.471585 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.471554 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v" (OuterVolumeSpecName: "kube-api-access-87n9v") pod "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" (UID: "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7"). InnerVolumeSpecName "kube-api-access-87n9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:13:15.473896 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.473867 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util" (OuterVolumeSpecName: "util") pod "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" (UID: "9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:13:15.570626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.570586 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87n9v\" (UniqueName: \"kubernetes.io/projected/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-kube-api-access-87n9v\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:13:15.570626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.570619 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:13:15.570626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:15.570628 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:13:16.201465 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:16.201416 2565 scope.go:117] "RemoveContainer" containerID="2852d0828a527c1e3d038f50ff0e801d4e48ead05f217caacb759d27f9a67447" Sep 16 19:13:16.330610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:16.330567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" event={"ID":"9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7","Type":"ContainerDied","Data":"86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4"} Sep 16 19:13:16.330610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:16.330600 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frp9jf" Sep 16 19:13:16.330846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:16.330625 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e6de1a06e7ab5f473c957adb17301eae7119890867711b5d9fd32915955aa4" Sep 16 19:13:21.298627 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.298588 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb"] Sep 16 19:13:21.299089 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299063 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="pull" Sep 16 19:13:21.299089 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299078 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="pull" Sep 16 19:13:21.299197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299098 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="util" Sep 16 19:13:21.299197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299103 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="util" Sep 16 19:13:21.299197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299116 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="extract" Sep 16 19:13:21.299197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299122 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="extract" Sep 16 19:13:21.299197 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.299195 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e91518d-0cfb-4bd1-bd7b-5421fdf32fd7" containerName="extract" Sep 16 19:13:21.304778 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.304756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.308779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.308760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6q2vc\"" Sep 16 19:13:21.309074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.309055 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 19:13:21.309074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.309067 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 19:13:21.415883 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.415849 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb"] Sep 16 19:13:21.418251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.418224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c328a311-dff1-4e95-a032-ec92725406f2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.418419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.418327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwjp\" (UniqueName: \"kubernetes.io/projected/c328a311-dff1-4e95-a032-ec92725406f2-kube-api-access-bjwjp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.518768 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.518713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwjp\" (UniqueName: \"kubernetes.io/projected/c328a311-dff1-4e95-a032-ec92725406f2-kube-api-access-bjwjp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.518928 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.518819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c328a311-dff1-4e95-a032-ec92725406f2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.519216 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.519199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c328a311-dff1-4e95-a032-ec92725406f2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.546683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.546653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwjp\" (UniqueName: \"kubernetes.io/projected/c328a311-dff1-4e95-a032-ec92725406f2-kube-api-access-bjwjp\") pod \"openshift-lws-operator-fd8ccff4c-pnmgb\" (UID: \"c328a311-dff1-4e95-a032-ec92725406f2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.627935 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.627847 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" Sep 16 19:13:21.908522 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:21.908446 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb"] Sep 16 19:13:21.913569 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:13:21.913535 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc328a311_dff1_4e95_a032_ec92725406f2.slice/crio-15693af7272c276aa7236da39cf6e0ac0a2aa54b4a5ae5761dc01f1085e84b0f WatchSource:0}: Error finding container 15693af7272c276aa7236da39cf6e0ac0a2aa54b4a5ae5761dc01f1085e84b0f: Status 404 returned error can't find the container with id 15693af7272c276aa7236da39cf6e0ac0a2aa54b4a5ae5761dc01f1085e84b0f Sep 16 19:13:22.360118 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:22.360075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" event={"ID":"c328a311-dff1-4e95-a032-ec92725406f2","Type":"ContainerStarted","Data":"15693af7272c276aa7236da39cf6e0ac0a2aa54b4a5ae5761dc01f1085e84b0f"} Sep 16 19:13:24.373532 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:24.373493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" event={"ID":"c328a311-dff1-4e95-a032-ec92725406f2","Type":"ContainerStarted","Data":"e03b0562894128bb290299a546d7691b305aacf9dd977db119166d59eb69c3d5"} Sep 16 19:13:36.543695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.543637 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pnmgb" podStartSLOduration=13.675758131 podStartE2EDuration="15.543620915s" totalImagesPullingTime="1.867862784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:24.413973407 +0000 UTC m=+488.619929159" watchObservedRunningTime="2026-09-16 19:13:36.543620915 +0000 UTC m=+500.749576667" Sep 16 19:13:36.545320 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.545301 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:13:36.548842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.548827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.553098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.553078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 19:13:36.553211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.553123 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 19:13:36.553211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.553149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-brh4c\"" Sep 16 19:13:36.553497 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.553483 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 19:13:36.553575 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.553488 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 19:13:36.576733 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.576705 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:13:36.642184 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642184 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtbr\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.642389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.642351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.742944 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.742912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.742944 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.742953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.742975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.743022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.743037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.743079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtbr\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.743106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.743845 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.743738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.745659 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.745637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.745763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.745720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.745806 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.745764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.745806 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.745784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.756727 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.756703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.756901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.756885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtbr\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr\") pod \"istiod-openshift-gateway-7b7995d5cd-cmflr\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:36.858031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:36.857947 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:37.157626 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:37.157549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:13:37.163335 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:13:37.163306 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80d603_0e2c_4f21_a3be_8c6ebc600ef4.slice/crio-330a0774bab8bfee319ec976baafdc5858235df8c01d28e5c6ab9ad60b229642 WatchSource:0}: Error finding container 330a0774bab8bfee319ec976baafdc5858235df8c01d28e5c6ab9ad60b229642: Status 404 returned error can't find the container with id 330a0774bab8bfee319ec976baafdc5858235df8c01d28e5c6ab9ad60b229642 Sep 16 19:13:37.438318 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:37.438216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" event={"ID":"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4","Type":"ContainerStarted","Data":"330a0774bab8bfee319ec976baafdc5858235df8c01d28e5c6ab9ad60b229642"} Sep 16 19:13:40.012472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:40.012426 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 19:13:40.012737 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:40.012507 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 16 19:13:40.461688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:40.461604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" event={"ID":"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4","Type":"ContainerStarted","Data":"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada"} Sep 16 19:13:40.461842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:40.461720 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:13:40.515387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:40.515321 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" podStartSLOduration=1.668517984 podStartE2EDuration="4.515299312s" totalImagesPullingTime="2.846781328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:40.51279321 +0000 UTC m=+504.718748974" watchObservedRunningTime="2026-09-16 19:13:40.515299312 +0000 UTC m=+504.721255059" Sep 16 19:13:41.468532 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:41.468498 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cmflr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 19:13:41.468909 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:41.468563 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:13:44.468108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:13:44.468074 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:14:00.224731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.224698 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t"] Sep 16 19:14:00.232821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.232801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.236576 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.236556 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 19:14:00.237336 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.237317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b6684\"" Sep 16 19:14:00.237438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.237335 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 19:14:00.258094 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.258068 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t"] Sep 16 19:14:00.346937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.346903 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb"] Sep 16 19:14:00.350808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.350791 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.352051 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.352024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92rk\" (UniqueName: \"kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.352163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.352062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.352204 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.352175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.376414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.376360 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb"] Sep 16 19:14:00.452684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb27m\" (UniqueName: \"kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.452684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.452894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.452894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.452894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92rk\" (UniqueName: \"kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.452894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.452885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.453111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.453096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.453167 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.453150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.466910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.466885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92rk\" (UniqueName: \"kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.505678 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.505590 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv"] Sep 16 19:14:00.509675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.509653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.541648 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.541609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:00.544769 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.544742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv"] Sep 16 19:14:00.553566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.553517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb27m\" (UniqueName: \"kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.553696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.553595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.553696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.553635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.554066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.554030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.554169 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.554096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.575130 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.575093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb27m\" (UniqueName: \"kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.655043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.654991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zpj\" (UniqueName: \"kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.655203 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.655085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.655203 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.655128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.659960 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.659940 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:00.707660 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.707620 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn"] Sep 16 19:14:00.717296 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.716669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.741860 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.741833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn"] Sep 16 19:14:00.755789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.755789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwlml\" (UniqueName: \"kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.755939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zpj\" (UniqueName: \"kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.755939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.755939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.755939 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.755920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.756093 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.756069 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.756155 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.756140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.784840 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.784811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zpj\" (UniqueName: \"kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.818719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.818684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:00.839580 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.839558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t"] Sep 16 19:14:00.841843 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:00.841811 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0d71ed_8e21_4e9a_a0ab_bf229723e0aa.slice/crio-c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3 WatchSource:0}: Error finding container c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3: Status 404 returned error can't find the container with id c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3 Sep 16 19:14:00.856625 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.856593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.856773 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.856687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwlml\" (UniqueName: \"kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.856773 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.856741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.857075 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.857048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.857165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.857121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.869049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.868893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwlml\" (UniqueName: \"kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:00.949211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:00.949186 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb"] Sep 16 19:14:00.950464 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:00.950433 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1670585_8ad3_4970_a2ad_73c3e084c708.slice/crio-dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211 WatchSource:0}: Error finding container dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211: Status 404 returned error can't find the container with id dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211 Sep 16 19:14:01.031405 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.031381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:01.129023 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.129000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv"] Sep 16 19:14:01.157296 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:01.157242 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53fbb63_bf00_4ce0_a1f8_b3202e2843d3.slice/crio-f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d WatchSource:0}: Error finding container f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d: Status 404 returned error can't find the container with id f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d Sep 16 19:14:01.332157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.332070 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn"] Sep 16 19:14:01.338679 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:01.338656 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf7a0ab_5c65_48cc_a2d5_4542394aad4e.slice/crio-b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83 WatchSource:0}: Error finding container b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83: Status 404 returned error can't find the container with id b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83 Sep 16 19:14:01.565639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.565604 2565 generic.go:356] "Generic (PLEG): container finished" podID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerID="2fd51874dc3c95ea92e9d559ef00e6b2bdc852b2f351508b1cec4249151aebfb" exitCode=0 Sep 16 19:14:01.565830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.565700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" event={"ID":"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa","Type":"ContainerDied","Data":"2fd51874dc3c95ea92e9d559ef00e6b2bdc852b2f351508b1cec4249151aebfb"} Sep 16 19:14:01.565830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.565757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" event={"ID":"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa","Type":"ContainerStarted","Data":"c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3"} Sep 16 19:14:01.567299 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.567261 2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerID="3eb0620ee16d61c3a9490c136f4a2e5decc78ff0daaeb7016e4073f8dabec799" exitCode=0 Sep 16 19:14:01.567385 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.567351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" event={"ID":"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e","Type":"ContainerDied","Data":"3eb0620ee16d61c3a9490c136f4a2e5decc78ff0daaeb7016e4073f8dabec799"} Sep 16 19:14:01.567440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.567401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" event={"ID":"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e","Type":"ContainerStarted","Data":"b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83"} Sep 16 19:14:01.568985 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.568969 2565 generic.go:356] "Generic (PLEG): container finished" podID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerID="0dc7eca8bec625d1adc06ebd046f706ecc8490b63c76b6006233b5d7276b7e63" exitCode=0 Sep 16 19:14:01.569081 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.569064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" event={"ID":"f1670585-8ad3-4970-a2ad-73c3e084c708","Type":"ContainerDied","Data":"0dc7eca8bec625d1adc06ebd046f706ecc8490b63c76b6006233b5d7276b7e63"} Sep 16 19:14:01.569146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.569112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" event={"ID":"f1670585-8ad3-4970-a2ad-73c3e084c708","Type":"ContainerStarted","Data":"dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211"} Sep 16 19:14:01.570743 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.570717 2565 generic.go:356] "Generic (PLEG): container finished" podID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerID="70cdc76ca0b2afc490bb1e67fe0ef95ac113db2eeab9a0acd111707f6689b3a9" exitCode=0 Sep 16 19:14:01.570831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.570755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" event={"ID":"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3","Type":"ContainerDied","Data":"70cdc76ca0b2afc490bb1e67fe0ef95ac113db2eeab9a0acd111707f6689b3a9"} Sep 16 19:14:01.570831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:01.570787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" event={"ID":"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3","Type":"ContainerStarted","Data":"f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d"} Sep 16 19:14:02.577805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:02.577724 2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerID="2680f24ff08e3cfe5df0b73e7fc63faf85562be4005bfaae756c8106aefdac2a" exitCode=0 Sep 16 19:14:02.578188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:02.577805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" event={"ID":"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e","Type":"ContainerDied","Data":"2680f24ff08e3cfe5df0b73e7fc63faf85562be4005bfaae756c8106aefdac2a"} Sep 16 19:14:03.588650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.588618 2565 generic.go:356] "Generic (PLEG): container finished" podID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerID="92f4ea6c1d8c59c5d7c8313323e5f5e34736252797b176127ab0b84815f2dd00" exitCode=0 Sep 16 19:14:03.589088 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.588704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" event={"ID":"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa","Type":"ContainerDied","Data":"92f4ea6c1d8c59c5d7c8313323e5f5e34736252797b176127ab0b84815f2dd00"} Sep 16 19:14:03.591140 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.591121 2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerID="c48c5b8f8147270e8526163c8b5b62f117159ae0d53fa5573119bbb30e48f2ee" exitCode=0 Sep 16 19:14:03.591235 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.591200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" event={"ID":"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e","Type":"ContainerDied","Data":"c48c5b8f8147270e8526163c8b5b62f117159ae0d53fa5573119bbb30e48f2ee"} Sep 16 19:14:03.592865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.592838 2565 generic.go:356] "Generic (PLEG): container finished" podID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerID="00b9cd0397ce80bede18125907f27d7960eb4b9f69c4b22228491ed2705bdb89" exitCode=0 Sep 16 19:14:03.592952 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.592922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" event={"ID":"f1670585-8ad3-4970-a2ad-73c3e084c708","Type":"ContainerDied","Data":"00b9cd0397ce80bede18125907f27d7960eb4b9f69c4b22228491ed2705bdb89"} Sep 16 19:14:03.594927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.594907 2565 generic.go:356] "Generic (PLEG): container finished" podID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerID="57fc747d3e45976389557efcdd6ec26786050873ce41391da1af68c40c8536c9" exitCode=0 Sep 16 19:14:03.595047 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:03.594949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" event={"ID":"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3","Type":"ContainerDied","Data":"57fc747d3e45976389557efcdd6ec26786050873ce41391da1af68c40c8536c9"} Sep 16 19:14:04.601983 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.601947 2565 generic.go:356] "Generic (PLEG): container finished" podID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerID="3563b0c4aff32ed2ed2627d1204e7c9d3e69af7e541e33b2062b5580049b38dc" exitCode=0 Sep 16 19:14:04.602431 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.602000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" event={"ID":"f1670585-8ad3-4970-a2ad-73c3e084c708","Type":"ContainerDied","Data":"3563b0c4aff32ed2ed2627d1204e7c9d3e69af7e541e33b2062b5580049b38dc"} Sep 16 19:14:04.604159 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.604133 2565 generic.go:356] "Generic (PLEG): container finished" podID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerID="c7c382f23a6f8c81df7054e9d6a0813661ed1c68429b0eea06f0696f4ac58e7c" exitCode=0 Sep 16 19:14:04.604273 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.604224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" event={"ID":"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3","Type":"ContainerDied","Data":"c7c382f23a6f8c81df7054e9d6a0813661ed1c68429b0eea06f0696f4ac58e7c"} Sep 16 19:14:04.606255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.606232 2565 generic.go:356] "Generic (PLEG): container finished" podID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerID="95310b8d8ef6e3eeae694abd5b6a99f9209a18a40923edd5ef9d13adaee5c8b8" exitCode=0 Sep 16 19:14:04.606366 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.606311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" event={"ID":"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa","Type":"ContainerDied","Data":"95310b8d8ef6e3eeae694abd5b6a99f9209a18a40923edd5ef9d13adaee5c8b8"} Sep 16 19:14:04.738074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.738044 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:04.783583 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.783555 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util\") pod \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " Sep 16 19:14:04.783752 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.783632 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle\") pod \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " Sep 16 19:14:04.783752 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.783675 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwlml\" (UniqueName: \"kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml\") pod \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\" (UID: \"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e\") " Sep 16 19:14:04.784377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.784326 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle" (OuterVolumeSpecName: "bundle") pod "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" (UID: "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:04.785996 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.785971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml" (OuterVolumeSpecName: "kube-api-access-jwlml") pod "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" (UID: "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e"). InnerVolumeSpecName "kube-api-access-jwlml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:04.789041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.789009 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util" (OuterVolumeSpecName: "util") pod "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" (UID: "ecf7a0ab-5c65-48cc-a2d5-4542394aad4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:04.884603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.884510 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:04.884603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.884543 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwlml\" (UniqueName: \"kubernetes.io/projected/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-kube-api-access-jwlml\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:04.884603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:04.884559 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecf7a0ab-5c65-48cc-a2d5-4542394aad4e-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.614188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.614155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" Sep 16 19:14:05.614188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.614185 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awtcgn" event={"ID":"ecf7a0ab-5c65-48cc-a2d5-4542394aad4e","Type":"ContainerDied","Data":"b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83"} Sep 16 19:14:05.614698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.614228 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b21cf9794427eb3f9d6f006b6bb9cecd19dcd6e14db71e8975b71c39695d0d83" Sep 16 19:14:05.759099 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.759065 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:05.789962 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.789938 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c92rk\" (UniqueName: \"kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk\") pod \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " Sep 16 19:14:05.790063 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.789994 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle\") pod \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " Sep 16 19:14:05.790117 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.790101 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util\") pod \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\" (UID: \"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa\") " Sep 16 19:14:05.790378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.790355 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle" (OuterVolumeSpecName: "bundle") pod "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" (UID: "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.792310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.792268 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk" (OuterVolumeSpecName: "kube-api-access-c92rk") pod "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" (UID: "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa"). InnerVolumeSpecName "kube-api-access-c92rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:05.799449 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.799421 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util" (OuterVolumeSpecName: "util") pod "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" (UID: "5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.812369 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.812336 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:05.815362 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.815343 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:05.890857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.890784 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle\") pod \"f1670585-8ad3-4970-a2ad-73c3e084c708\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " Sep 16 19:14:05.890857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.890835 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util\") pod \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " Sep 16 19:14:05.891045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.890904 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb27m\" (UniqueName: \"kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m\") pod \"f1670585-8ad3-4970-a2ad-73c3e084c708\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " Sep 16 19:14:05.891045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.890935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util\") pod \"f1670585-8ad3-4970-a2ad-73c3e084c708\" (UID: \"f1670585-8ad3-4970-a2ad-73c3e084c708\") " Sep 16 19:14:05.891045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.890962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle\") pod \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " Sep 16 19:14:05.891199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891081 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5zpj\" (UniqueName: \"kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj\") pod \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\" (UID: \"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3\") " Sep 16 19:14:05.891637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891361 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.891637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891367 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle" (OuterVolumeSpecName: "bundle") pod "f1670585-8ad3-4970-a2ad-73c3e084c708" (UID: "f1670585-8ad3-4970-a2ad-73c3e084c708"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.891637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891386 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92rk\" (UniqueName: \"kubernetes.io/projected/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-kube-api-access-c92rk\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.891637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891403 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.891637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.891603 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle" (OuterVolumeSpecName: "bundle") pod "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" (UID: "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.893536 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.893501 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj" (OuterVolumeSpecName: "kube-api-access-z5zpj") pod "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" (UID: "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3"). InnerVolumeSpecName "kube-api-access-z5zpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:05.894022 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.894002 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m" (OuterVolumeSpecName: "kube-api-access-jb27m") pod "f1670585-8ad3-4970-a2ad-73c3e084c708" (UID: "f1670585-8ad3-4970-a2ad-73c3e084c708"). InnerVolumeSpecName "kube-api-access-jb27m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:05.896433 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.896415 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util" (OuterVolumeSpecName: "util") pod "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" (UID: "a53fbb63-bf00-4ce0-a1f8-b3202e2843d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.897829 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.897808 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util" (OuterVolumeSpecName: "util") pod "f1670585-8ad3-4970-a2ad-73c3e084c708" (UID: "f1670585-8ad3-4970-a2ad-73c3e084c708"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:05.992201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992163 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.992201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992192 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.992201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992202 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb27m\" (UniqueName: \"kubernetes.io/projected/f1670585-8ad3-4970-a2ad-73c3e084c708-kube-api-access-jb27m\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.992201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992211 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1670585-8ad3-4970-a2ad-73c3e084c708-util\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.992503 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992221 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:05.992503 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:05.992229 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5zpj\" (UniqueName: \"kubernetes.io/projected/a53fbb63-bf00-4ce0-a1f8-b3202e2843d3-kube-api-access-z5zpj\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:14:06.620688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.620592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" event={"ID":"5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa","Type":"ContainerDied","Data":"c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3"} Sep 16 19:14:06.620688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.620622 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6czh5t" Sep 16 19:14:06.620688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.620677 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d04836730b74b822dcecb2bc61e092b59e28eca0eb8fd47914e8c40aa554e3" Sep 16 19:14:06.622552 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.622518 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" Sep 16 19:14:06.622552 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.622532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p8clb" event={"ID":"f1670585-8ad3-4970-a2ad-73c3e084c708","Type":"ContainerDied","Data":"dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211"} Sep 16 19:14:06.622732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.622574 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe35baa89c957ed102999083372858ab54da163373c7b04229c6e87c31e9211" Sep 16 19:14:06.625091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.625073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" event={"ID":"a53fbb63-bf00-4ce0-a1f8-b3202e2843d3","Type":"ContainerDied","Data":"f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d"} Sep 16 19:14:06.625091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.625090 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9mhwv" Sep 16 19:14:06.625200 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:06.625100 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5117cc27278e34590812986908075b8da407951a031ea0bcc9625bc2f88b92d" Sep 16 19:14:11.699659 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699579 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f"] Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699931 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699941 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699951 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699956 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699964 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699970 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699977 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699983 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699992 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.699998 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700003 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700009 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700016 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700020 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700025 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700031 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="util" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700038 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700042 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700052 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700057 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700065 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700069 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="pull" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700074 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="extract" Sep 16 19:14:11.700101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700078 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="extract" Sep 16 19:14:11.700838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700128 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53fbb63-bf00-4ce0-a1f8-b3202e2843d3" containerName="extract" Sep 16 19:14:11.700838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700140 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf7a0ab-5c65-48cc-a2d5-4542394aad4e" containerName="extract" Sep 16 19:14:11.700838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700145 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0d71ed-8e21-4e9a-a0ab-bf229723e0aa" containerName="extract" Sep 16 19:14:11.700838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.700151 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1670585-8ad3-4970-a2ad-73c3e084c708" containerName="extract" Sep 16 19:14:11.708793 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.708770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:11.712743 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.712723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 19:14:11.713518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.713500 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mhmcp\"" Sep 16 19:14:11.713620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.713518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 19:14:11.728014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.727986 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f"] Sep 16 19:14:11.743886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.743856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5586x\" (UniqueName: \"kubernetes.io/projected/f9ce13d9-b9f4-42f4-874f-1d71c55dfb10-kube-api-access-5586x\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qhg5f\" (UID: \"f9ce13d9-b9f4-42f4-874f-1d71c55dfb10\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:11.844696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.844663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5586x\" (UniqueName: \"kubernetes.io/projected/f9ce13d9-b9f4-42f4-874f-1d71c55dfb10-kube-api-access-5586x\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qhg5f\" (UID: \"f9ce13d9-b9f4-42f4-874f-1d71c55dfb10\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:11.869665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:11.869635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5586x\" (UniqueName: \"kubernetes.io/projected/f9ce13d9-b9f4-42f4-874f-1d71c55dfb10-kube-api-access-5586x\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qhg5f\" (UID: \"f9ce13d9-b9f4-42f4-874f-1d71c55dfb10\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:12.018758 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:12.018720 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:12.325002 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:12.324925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f"] Sep 16 19:14:12.328554 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:12.328524 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ce13d9_b9f4_42f4_874f_1d71c55dfb10.slice/crio-feda7f12005b3a4fea067a230d48b1f9804a20e4e6f2004118bf63d0eaca386b WatchSource:0}: Error finding container feda7f12005b3a4fea067a230d48b1f9804a20e4e6f2004118bf63d0eaca386b: Status 404 returned error can't find the container with id feda7f12005b3a4fea067a230d48b1f9804a20e4e6f2004118bf63d0eaca386b Sep 16 19:14:12.657767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:12.657673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" event={"ID":"f9ce13d9-b9f4-42f4-874f-1d71c55dfb10","Type":"ContainerStarted","Data":"feda7f12005b3a4fea067a230d48b1f9804a20e4e6f2004118bf63d0eaca386b"} Sep 16 19:14:14.671183 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:14.671144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" event={"ID":"f9ce13d9-b9f4-42f4-874f-1d71c55dfb10","Type":"ContainerStarted","Data":"e2c63740f39c33c40333b045b516529be49b33e2b94b0d0a05a127e3c55b9d95"} Sep 16 19:14:14.671609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:14.671272 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:14.795949 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:14.795893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" podStartSLOduration=2.065135977 podStartE2EDuration="3.795876591s" totalImagesPullingTime="1.730740614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:14.784425347 +0000 UTC m=+538.990381099" watchObservedRunningTime="2026-09-16 19:14:14.795876591 +0000 UTC m=+539.001832343" Sep 16 19:14:16.689499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.689467 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f"] Sep 16 19:14:16.693326 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.693302 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.724139 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.724114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mqbrx\"" Sep 16 19:14:16.784404 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.784369 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f"] Sep 16 19:14:16.786936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.786913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s82k\" (UniqueName: \"kubernetes.io/projected/f147649b-57d6-4b35-85b7-3a5a5db2f626-kube-api-access-7s82k\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.787033 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.786946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f147649b-57d6-4b35-85b7-3a5a5db2f626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.887869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.887830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s82k\" (UniqueName: \"kubernetes.io/projected/f147649b-57d6-4b35-85b7-3a5a5db2f626-kube-api-access-7s82k\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.887869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.887871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f147649b-57d6-4b35-85b7-3a5a5db2f626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.888342 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.888324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f147649b-57d6-4b35-85b7-3a5a5db2f626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:16.902924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:16.902896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s82k\" (UniqueName: \"kubernetes.io/projected/f147649b-57d6-4b35-85b7-3a5a5db2f626-kube-api-access-7s82k\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vx52f\" (UID: \"f147649b-57d6-4b35-85b7-3a5a5db2f626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:17.003718 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:17.003685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:17.341821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:17.341729 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f"] Sep 16 19:14:17.346500 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:14:17.346464 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf147649b_57d6_4b35_85b7_3a5a5db2f626.slice/crio-e43070bda02ae8abcce1e283deca4b87dd979d1bb8ee6ee040ac1429c98dbf0c WatchSource:0}: Error finding container e43070bda02ae8abcce1e283deca4b87dd979d1bb8ee6ee040ac1429c98dbf0c: Status 404 returned error can't find the container with id e43070bda02ae8abcce1e283deca4b87dd979d1bb8ee6ee040ac1429c98dbf0c Sep 16 19:14:17.696118 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:17.696036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" event={"ID":"f147649b-57d6-4b35-85b7-3a5a5db2f626","Type":"ContainerStarted","Data":"e43070bda02ae8abcce1e283deca4b87dd979d1bb8ee6ee040ac1429c98dbf0c"} Sep 16 19:14:21.720942 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:21.720900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" event={"ID":"f147649b-57d6-4b35-85b7-3a5a5db2f626","Type":"ContainerStarted","Data":"422d6722db7b7cf2898d36986b37b1a4ef5f28a6f571408281b7556ddf1ed937"} Sep 16 19:14:21.721470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:21.721002 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:14:21.750350 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:21.750297 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" podStartSLOduration=1.954482809 podStartE2EDuration="5.75026234s" totalImagesPullingTime="3.795779531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:21.748199115 +0000 UTC m=+545.954154868" watchObservedRunningTime="2026-09-16 19:14:21.75026234 +0000 UTC m=+545.956218093" Sep 16 19:14:25.679436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:25.679401 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qhg5f" Sep 16 19:14:32.729689 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:14:32.729656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vx52f" Sep 16 19:15:05.970932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:05.970897 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xpwcg"] Sep 16 19:15:05.974066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:05.974049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:05.977037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:05.977018 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7wrrl\"" Sep 16 19:15:05.977427 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:05.977415 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 19:15:06.000644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.000612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xpwcg"] Sep 16 19:15:06.029638 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.029604 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xpwcg"] Sep 16 19:15:06.120255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.120224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c1c8572-4101-4733-8d39-83b2a1f1260d-config-file\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.120438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.120272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdd6\" (UniqueName: \"kubernetes.io/projected/1c1c8572-4101-4733-8d39-83b2a1f1260d-kube-api-access-9kdd6\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.220816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.220777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c1c8572-4101-4733-8d39-83b2a1f1260d-config-file\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.221021 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.220830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdd6\" (UniqueName: \"kubernetes.io/projected/1c1c8572-4101-4733-8d39-83b2a1f1260d-kube-api-access-9kdd6\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.221437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.221417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1c1c8572-4101-4733-8d39-83b2a1f1260d-config-file\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.233529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.233507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdd6\" (UniqueName: \"kubernetes.io/projected/1c1c8572-4101-4733-8d39-83b2a1f1260d-kube-api-access-9kdd6\") pod \"limitador-limitador-84994f7ffc-xpwcg\" (UID: \"1c1c8572-4101-4733-8d39-83b2a1f1260d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.284057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.284026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:06.580269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.580185 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-xpwcg"] Sep 16 19:15:06.583602 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:15:06.583569 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1c8572_4101_4733_8d39_83b2a1f1260d.slice/crio-30187f76272158fecd6780385eff8d8f3f1d566f0bfb64c4ef0d91427cee485f WatchSource:0}: Error finding container 30187f76272158fecd6780385eff8d8f3f1d566f0bfb64c4ef0d91427cee485f: Status 404 returned error can't find the container with id 30187f76272158fecd6780385eff8d8f3f1d566f0bfb64c4ef0d91427cee485f Sep 16 19:15:06.971565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:06.971531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" event={"ID":"1c1c8572-4101-4733-8d39-83b2a1f1260d","Type":"ContainerStarted","Data":"30187f76272158fecd6780385eff8d8f3f1d566f0bfb64c4ef0d91427cee485f"} Sep 16 19:15:08.983493 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:08.983397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" event={"ID":"1c1c8572-4101-4733-8d39-83b2a1f1260d","Type":"ContainerStarted","Data":"8827a02a134feb4a18c9de3355ab792786730fce04f531bcd3bc1a1399a1e0c0"} Sep 16 19:15:08.983943 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:08.983536 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:09.019400 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:09.019352 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" podStartSLOduration=1.876843426 podStartE2EDuration="4.019335878s" totalImagesPullingTime="2.142492452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:09.017084603 +0000 UTC m=+593.223040355" watchObservedRunningTime="2026-09-16 19:15:09.019335878 +0000 UTC m=+593.225291631" Sep 16 19:15:19.990131 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:19.990093 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-xpwcg" Sep 16 19:15:59.484596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.484547 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:15:59.485150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.484758 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerName="discovery" containerID="cri-o://b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada" gracePeriod=30 Sep 16 19:15:59.731146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.731118 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:15:59.887473 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjtbr\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887473 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887559 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887596 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887622 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887665 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token\") pod \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\" (UID: \"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4\") " Sep 16 19:15:59.887911 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887880 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:15:59.888015 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.887993 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-ca-configmap\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.890107 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890066 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs" (OuterVolumeSpecName: "local-certs") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:15:59.890255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:15:59.890351 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890331 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts" (OuterVolumeSpecName: "cacerts") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:15:59.890480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr" (OuterVolumeSpecName: "kube-api-access-xjtbr") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "kube-api-access-xjtbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:15:59.890598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:15:59.890598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.890533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" (UID: "ee80d603-0e2c-4f21-a3be-8c6ebc600ef4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:15:59.988736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988701 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-cacerts\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.988736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988728 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-local-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.988736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988739 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-csr-dns-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.988965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988748 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-kubeconfig\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.988965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988756 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:15:59.988965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:15:59.988764 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtbr\" (UniqueName: \"kubernetes.io/projected/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4-kube-api-access-xjtbr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:16:00.264288 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.264259 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerID="b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada" exitCode=0 Sep 16 19:16:00.264436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.264329 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" Sep 16 19:16:00.264436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.264346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" event={"ID":"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4","Type":"ContainerDied","Data":"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada"} Sep 16 19:16:00.264436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.264397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr" event={"ID":"ee80d603-0e2c-4f21-a3be-8c6ebc600ef4","Type":"ContainerDied","Data":"330a0774bab8bfee319ec976baafdc5858235df8c01d28e5c6ab9ad60b229642"} Sep 16 19:16:00.264436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.264417 2565 scope.go:117] "RemoveContainer" containerID="b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada" Sep 16 19:16:00.273084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.273067 2565 scope.go:117] "RemoveContainer" containerID="b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada" Sep 16 19:16:00.273379 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:16:00.273360 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada\": container with ID starting with b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada not found: ID does not exist" containerID="b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada" Sep 16 19:16:00.273447 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.273388 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada"} err="failed to get container status \"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada\": rpc error: code = NotFound desc = could not find container \"b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada\": container with ID starting with b4c6d06fbe929de4fe26d118e38d9a6f7a74992910c25ce87fd8a0e9fc864ada not found: ID does not exist" Sep 16 19:16:00.295598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.295565 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:16:00.299711 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:00.299689 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cmflr"] Sep 16 19:16:02.253208 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:02.253174 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" path="/var/lib/kubelet/pods/ee80d603-0e2c-4f21-a3be-8c6ebc600ef4/volumes" Sep 16 19:16:14.454927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.454887 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:16:14.455513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.455497 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerName="discovery" Sep 16 19:16:14.455562 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.455516 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerName="discovery" Sep 16 19:16:14.455616 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.455605 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee80d603-0e2c-4f21-a3be-8c6ebc600ef4" containerName="discovery" Sep 16 19:16:14.458888 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.458869 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.463705 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.463684 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 19:16:14.464449 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.464426 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 19:16:14.464568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.464446 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 19:16:14.464568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.464446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-47gpw\"" Sep 16 19:16:14.474688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.474664 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-776f74ff6f-6pndf"] Sep 16 19:16:14.477975 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.477959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.481079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.481058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 16 19:16:14.481164 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.481111 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rvjh4\"" Sep 16 19:16:14.489231 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.489209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:16:14.510318 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.510263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s8v\" (UniqueName: \"kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.510435 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.510352 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-776f74ff6f-6pndf"] Sep 16 19:16:14.510435 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.510416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.510534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.510442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.510534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.510479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrsj6\" (UniqueName: \"kubernetes.io/projected/575fda9f-5e01-4324-86e5-605727d99fd5-kube-api-access-mrsj6\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.556261 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.556228 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vd66v"] Sep 16 19:16:14.559651 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.559635 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.562457 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.562433 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-sjxhm\"" Sep 16 19:16:14.563166 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.563149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 19:16:14.575054 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.575027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vd66v"] Sep 16 19:16:14.610919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.610887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrsj6\" (UniqueName: \"kubernetes.io/projected/575fda9f-5e01-4324-86e5-605727d99fd5-kube-api-access-mrsj6\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.610919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.610926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s8v\" (UniqueName: \"kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.611160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.610963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbws\" (UniqueName: \"kubernetes.io/projected/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-kube-api-access-mpbws\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.611160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.611003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-data\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.611160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.611023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.611160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.611086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.611401 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:16:14.611210 2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 19:16:14.611401 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:16:14.611327 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert podName:575fda9f-5e01-4324-86e5-605727d99fd5 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:15.111275482 +0000 UTC m=+659.317231224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert") pod "llmisvc-controller-manager-776f74ff6f-6pndf" (UID: "575fda9f-5e01-4324-86e5-605727d99fd5") : secret "llmisvc-webhook-server-cert" not found Sep 16 19:16:14.613590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.613569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.624082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.624049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrsj6\" (UniqueName: \"kubernetes.io/projected/575fda9f-5e01-4324-86e5-605727d99fd5-kube-api-access-mrsj6\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:14.626001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.625983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s8v\" (UniqueName: \"kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v\") pod \"kserve-controller-manager-58dc55f4b7-plgnw\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.712155 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.712056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbws\" (UniqueName: \"kubernetes.io/projected/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-kube-api-access-mpbws\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.712155 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.712125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-data\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.712568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.712547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-data\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.750986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.750950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbws\" (UniqueName: \"kubernetes.io/projected/490a708e-ac96-4ed3-9aeb-4024ddf1ec2d-kube-api-access-mpbws\") pod \"seaweedfs-64568bcd49-vd66v\" (UID: \"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d\") " pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:14.770827 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.770796 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:14.869633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:14.869600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:15.059656 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.058372 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:16:15.061593 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:16:15.061565 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae7c44f_84ca_4b9d_8f19_fedf4182ed62.slice/crio-df488a1d62287fd04c3aabf6c0490f4ea5b243584485350bf743ce60d7f09f65 WatchSource:0}: Error finding container df488a1d62287fd04c3aabf6c0490f4ea5b243584485350bf743ce60d7f09f65: Status 404 returned error can't find the container with id df488a1d62287fd04c3aabf6c0490f4ea5b243584485350bf743ce60d7f09f65 Sep 16 19:16:15.062995 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.062977 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:16:15.116610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.116563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:15.116760 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:16:15.116712 2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 16 19:16:15.116807 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:16:15.116775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert podName:575fda9f-5e01-4324-86e5-605727d99fd5 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:16.116758771 +0000 UTC m=+660.322714505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert") pod "llmisvc-controller-manager-776f74ff6f-6pndf" (UID: "575fda9f-5e01-4324-86e5-605727d99fd5") : secret "llmisvc-webhook-server-cert" not found Sep 16 19:16:15.149423 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.149397 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vd66v"] Sep 16 19:16:15.150847 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:16:15.150826 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490a708e_ac96_4ed3_9aeb_4024ddf1ec2d.slice/crio-d9fb17ec94d64fba9069b757e3b6c0b4bdf3b43bd711acf25e7ec34e2be73d46 WatchSource:0}: Error finding container d9fb17ec94d64fba9069b757e3b6c0b4bdf3b43bd711acf25e7ec34e2be73d46: Status 404 returned error can't find the container with id d9fb17ec94d64fba9069b757e3b6c0b4bdf3b43bd711acf25e7ec34e2be73d46 Sep 16 19:16:15.346209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.346098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" event={"ID":"fae7c44f-84ca-4b9d-8f19-fedf4182ed62","Type":"ContainerStarted","Data":"df488a1d62287fd04c3aabf6c0490f4ea5b243584485350bf743ce60d7f09f65"} Sep 16 19:16:15.347548 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:15.347519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vd66v" event={"ID":"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d","Type":"ContainerStarted","Data":"d9fb17ec94d64fba9069b757e3b6c0b4bdf3b43bd711acf25e7ec34e2be73d46"} Sep 16 19:16:16.128685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:16.128631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:16.133009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:16.132978 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575fda9f-5e01-4324-86e5-605727d99fd5-cert\") pod \"llmisvc-controller-manager-776f74ff6f-6pndf\" (UID: \"575fda9f-5e01-4324-86e5-605727d99fd5\") " pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:16.291654 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:16.291620 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rvjh4\"" Sep 16 19:16:16.298925 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:16.298895 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:16.603804 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:16.603705 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-776f74ff6f-6pndf"] Sep 16 19:16:17.261636 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:16:17.261603 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod575fda9f_5e01_4324_86e5_605727d99fd5.slice/crio-5a5f0bcfa31ce2cf39f36825115613761706d9ba320bbe9d8929b9ca757ff351 WatchSource:0}: Error finding container 5a5f0bcfa31ce2cf39f36825115613761706d9ba320bbe9d8929b9ca757ff351: Status 404 returned error can't find the container with id 5a5f0bcfa31ce2cf39f36825115613761706d9ba320bbe9d8929b9ca757ff351 Sep 16 19:16:17.367362 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:17.367316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" event={"ID":"575fda9f-5e01-4324-86e5-605727d99fd5","Type":"ContainerStarted","Data":"5a5f0bcfa31ce2cf39f36825115613761706d9ba320bbe9d8929b9ca757ff351"} Sep 16 19:16:18.964196 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:18.964173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 19:16:19.382274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:19.382159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vd66v" event={"ID":"490a708e-ac96-4ed3-9aeb-4024ddf1ec2d","Type":"ContainerStarted","Data":"8a4986bf3ae0b41f23c5414cd1e4b7a140fe1e8d424dd8cecefb84d13cd34001"} Sep 16 19:16:19.382274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:19.382227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:19.384690 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:19.384659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" event={"ID":"fae7c44f-84ca-4b9d-8f19-fedf4182ed62","Type":"ContainerStarted","Data":"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f"} Sep 16 19:16:19.384852 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:19.384831 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:19.460818 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:19.460758 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vd66v" podStartSLOduration=1.65188505 podStartE2EDuration="5.460740823s" totalImagesPullingTime="3.808855773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:19.423076507 +0000 UTC m=+663.629032260" watchObservedRunningTime="2026-09-16 19:16:19.460740823 +0000 UTC m=+663.666696575" Sep 16 19:16:21.396222 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:21.396188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" event={"ID":"575fda9f-5e01-4324-86e5-605727d99fd5","Type":"ContainerStarted","Data":"c70edd11eab62a88cdeec0daf7d53beee79972befdca3d63b229e1c98136607d"} Sep 16 19:16:21.396648 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:21.396330 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:16:21.421573 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:21.421524 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" podStartSLOduration=4.323453192 podStartE2EDuration="7.421508766s" totalImagesPullingTime="3.098055574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:21.421228173 +0000 UTC m=+665.627183926" watchObservedRunningTime="2026-09-16 19:16:21.421508766 +0000 UTC m=+665.627464556" Sep 16 19:16:21.422434 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:21.422398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" podStartSLOduration=3.669408551 podStartE2EDuration="7.422385909s" totalImagesPullingTime="3.752977358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:19.460824252 +0000 UTC m=+663.666780005" watchObservedRunningTime="2026-09-16 19:16:21.422385909 +0000 UTC m=+665.628341686" Sep 16 19:16:25.392486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:25.392454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vd66v" Sep 16 19:16:50.395436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:50.395403 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:16:52.404550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:16:52.404520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-776f74ff6f-6pndf" Sep 16 19:17:02.539788 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.539748 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:17:02.540397 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.539941 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" podUID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" containerName="manager" containerID="cri-o://5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f" gracePeriod=10 Sep 16 19:17:02.576770 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.576740 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-bkbbz"] Sep 16 19:17:02.579264 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.579248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.586878 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.586847 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-bkbbz"] Sep 16 19:17:02.643868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.643833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fcddc6a-143b-4386-a714-08b12044088a-cert\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.644024 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.643888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n46\" (UniqueName: \"kubernetes.io/projected/2fcddc6a-143b-4386-a714-08b12044088a-kube-api-access-l6n46\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.744925 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.744892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fcddc6a-143b-4386-a714-08b12044088a-cert\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.745079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.744939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n46\" (UniqueName: \"kubernetes.io/projected/2fcddc6a-143b-4386-a714-08b12044088a-kube-api-access-l6n46\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.747530 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.747507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fcddc6a-143b-4386-a714-08b12044088a-cert\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.758260 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.758240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n46\" (UniqueName: \"kubernetes.io/projected/2fcddc6a-143b-4386-a714-08b12044088a-kube-api-access-l6n46\") pod \"kserve-controller-manager-58dc55f4b7-bkbbz\" (UID: \"2fcddc6a-143b-4386-a714-08b12044088a\") " pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.778485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.778464 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:17:02.845456 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.845373 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9s8v\" (UniqueName: \"kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v\") pod \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " Sep 16 19:17:02.845456 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.845419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert\") pod \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\" (UID: \"fae7c44f-84ca-4b9d-8f19-fedf4182ed62\") " Sep 16 19:17:02.847847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.847818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert" (OuterVolumeSpecName: "cert") pod "fae7c44f-84ca-4b9d-8f19-fedf4182ed62" (UID: "fae7c44f-84ca-4b9d-8f19-fedf4182ed62"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:17:02.847847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.847831 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v" (OuterVolumeSpecName: "kube-api-access-t9s8v") pod "fae7c44f-84ca-4b9d-8f19-fedf4182ed62" (UID: "fae7c44f-84ca-4b9d-8f19-fedf4182ed62"). InnerVolumeSpecName "kube-api-access-t9s8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:17:02.921085 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.921045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:02.946399 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.946372 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9s8v\" (UniqueName: \"kubernetes.io/projected/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-kube-api-access-t9s8v\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:17:02.946399 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:02.946396 2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae7c44f-84ca-4b9d-8f19-fedf4182ed62-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:17:03.198347 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.198184 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-bkbbz"] Sep 16 19:17:03.200733 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:17:03.200699 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcddc6a_143b_4386_a714_08b12044088a.slice/crio-feed9e9c2b9fb16a41389e0adee4d5d1b10899a4e795c00b90d3bfb49cfa08cb WatchSource:0}: Error finding container feed9e9c2b9fb16a41389e0adee4d5d1b10899a4e795c00b90d3bfb49cfa08cb: Status 404 returned error can't find the container with id feed9e9c2b9fb16a41389e0adee4d5d1b10899a4e795c00b90d3bfb49cfa08cb Sep 16 19:17:03.625389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.625357 2565 generic.go:356] "Generic (PLEG): container finished" podID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" containerID="5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f" exitCode=0 Sep 16 19:17:03.625815 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.625400 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" Sep 16 19:17:03.625815 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.625436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" event={"ID":"fae7c44f-84ca-4b9d-8f19-fedf4182ed62","Type":"ContainerDied","Data":"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f"} Sep 16 19:17:03.625815 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.625488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-plgnw" event={"ID":"fae7c44f-84ca-4b9d-8f19-fedf4182ed62","Type":"ContainerDied","Data":"df488a1d62287fd04c3aabf6c0490f4ea5b243584485350bf743ce60d7f09f65"} Sep 16 19:17:03.625815 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.625511 2565 scope.go:117] "RemoveContainer" containerID="5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f" Sep 16 19:17:03.626931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.626886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" event={"ID":"2fcddc6a-143b-4386-a714-08b12044088a","Type":"ContainerStarted","Data":"feed9e9c2b9fb16a41389e0adee4d5d1b10899a4e795c00b90d3bfb49cfa08cb"} Sep 16 19:17:03.635713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.635697 2565 scope.go:117] "RemoveContainer" containerID="5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f" Sep 16 19:17:03.635978 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:17:03.635958 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f\": container with ID starting with 5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f not found: ID does not exist" containerID="5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f" Sep 16 19:17:03.636016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.635990 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f"} err="failed to get container status \"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f\": rpc error: code = NotFound desc = could not find container \"5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f\": container with ID starting with 5db500f45d74cf16c1c9ae4f1a18b3175b4f73e6f021def27d1d6a5d07d2276f not found: ID does not exist" Sep 16 19:17:03.653885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.653859 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:17:03.659775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:03.659741 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-58dc55f4b7-plgnw"] Sep 16 19:17:04.253553 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:04.253519 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" path="/var/lib/kubelet/pods/fae7c44f-84ca-4b9d-8f19-fedf4182ed62/volumes" Sep 16 19:17:04.634605 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:04.634514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" event={"ID":"2fcddc6a-143b-4386-a714-08b12044088a","Type":"ContainerStarted","Data":"24e24c8b73e6477e7fd0bf4a9cdc46d195ffbefd2e5127c33480e1050643fbe3"} Sep 16 19:17:04.634974 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:04.634653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:04.660204 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:04.660144 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" podStartSLOduration=2.189968039 podStartE2EDuration="2.660127052s" totalImagesPullingTime="470.159013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:04.657477198 +0000 UTC m=+708.863432964" watchObservedRunningTime="2026-09-16 19:17:04.660127052 +0000 UTC m=+708.866082805" Sep 16 19:17:35.645205 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:35.645174 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-58dc55f4b7-bkbbz" Sep 16 19:17:37.329091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.329054 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-cnxwn"] Sep 16 19:17:37.329730 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.329711 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" containerName="manager" Sep 16 19:17:37.329790 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.329733 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" containerName="manager" Sep 16 19:17:37.329854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.329843 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae7c44f-84ca-4b9d-8f19-fedf4182ed62" containerName="manager" Sep 16 19:17:37.333673 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.333653 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.337542 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.337520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 19:17:37.337652 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.337578 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-m8tkd\"" Sep 16 19:17:37.343805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.343782 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-bgs5l"] Sep 16 19:17:37.346980 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.346962 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.350510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.350491 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 16 19:17:37.350695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.350674 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-2ncvp\"" Sep 16 19:17:37.350907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.350885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 16 19:17:37.354927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.354908 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-cnxwn"] Sep 16 19:17:37.368069 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.368043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bgs5l"] Sep 16 19:17:37.448426 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.448386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jxr\" (UniqueName: \"kubernetes.io/projected/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-kube-api-access-z7jxr\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.448611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.448443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-metrics-certs\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.448611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.448516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-tls-certs\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.448611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.448594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-cert\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.448758 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.448672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2ct\" (UniqueName: \"kubernetes.io/projected/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-kube-api-access-hl2ct\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.549774 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.549743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2ct\" (UniqueName: \"kubernetes.io/projected/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-kube-api-access-hl2ct\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.549979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.549812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jxr\" (UniqueName: \"kubernetes.io/projected/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-kube-api-access-z7jxr\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.549979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.549839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-metrics-certs\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.549979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.549858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-tls-certs\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.549979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.549877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-cert\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.552530 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.552498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-tls-certs\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.552651 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.552531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-cert\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.552651 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.552583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-metrics-certs\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.562672 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.562646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2ct\" (UniqueName: \"kubernetes.io/projected/4696e711-30fe-42d0-b3ed-95a7bac3b5c2-kube-api-access-hl2ct\") pod \"model-serving-api-7b954ffc95-cnxwn\" (UID: \"4696e711-30fe-42d0-b3ed-95a7bac3b5c2\") " pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.567326 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.567305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jxr\" (UniqueName: \"kubernetes.io/projected/5d17a5f4-a60b-47ef-99c8-128f4a388f4e-kube-api-access-z7jxr\") pod \"odh-model-controller-7b966d7984-bgs5l\" (UID: \"5d17a5f4-a60b-47ef-99c8-128f4a388f4e\") " pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.645699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.645601 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:37.660060 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.660025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:37.935065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.935042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-cnxwn"] Sep 16 19:17:37.937158 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:17:37.937133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4696e711_30fe_42d0_b3ed_95a7bac3b5c2.slice/crio-782a07fe57e5325ba03337ddf52109225ee252a8a3f3ddaaea1c81b4d5268440 WatchSource:0}: Error finding container 782a07fe57e5325ba03337ddf52109225ee252a8a3f3ddaaea1c81b4d5268440: Status 404 returned error can't find the container with id 782a07fe57e5325ba03337ddf52109225ee252a8a3f3ddaaea1c81b4d5268440 Sep 16 19:17:37.951518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:37.951492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bgs5l"] Sep 16 19:17:37.953319 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:17:37.953274 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d17a5f4_a60b_47ef_99c8_128f4a388f4e.slice/crio-7408dfcfd6c1f1a3c7a02947a6d9667d3bb1b58c7d01d71f9221ffd1811e08f3 WatchSource:0}: Error finding container 7408dfcfd6c1f1a3c7a02947a6d9667d3bb1b58c7d01d71f9221ffd1811e08f3: Status 404 returned error can't find the container with id 7408dfcfd6c1f1a3c7a02947a6d9667d3bb1b58c7d01d71f9221ffd1811e08f3 Sep 16 19:17:38.829419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:38.829376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bgs5l" event={"ID":"5d17a5f4-a60b-47ef-99c8-128f4a388f4e","Type":"ContainerStarted","Data":"7408dfcfd6c1f1a3c7a02947a6d9667d3bb1b58c7d01d71f9221ffd1811e08f3"} Sep 16 19:17:38.833535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:38.833503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-cnxwn" event={"ID":"4696e711-30fe-42d0-b3ed-95a7bac3b5c2","Type":"ContainerStarted","Data":"782a07fe57e5325ba03337ddf52109225ee252a8a3f3ddaaea1c81b4d5268440"} Sep 16 19:17:41.854455 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.854421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bgs5l" event={"ID":"5d17a5f4-a60b-47ef-99c8-128f4a388f4e","Type":"ContainerStarted","Data":"a8d4890d6519b8a542d68f54232d5faef3338d081ac3d1ea79b8f104a1473ba2"} Sep 16 19:17:41.854865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.854485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:41.856025 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.855998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-cnxwn" event={"ID":"4696e711-30fe-42d0-b3ed-95a7bac3b5c2","Type":"ContainerStarted","Data":"33f3ea22a51ec3e726dba76bd1ef8276822f1d8322af8910f960be3fbe5b2c36"} Sep 16 19:17:41.856141 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.856110 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:41.876494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.876441 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-bgs5l" podStartSLOduration=1.141135342 podStartE2EDuration="4.87642439s" totalImagesPullingTime="3.735289048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:41.874850983 +0000 UTC m=+746.080806733" watchObservedRunningTime="2026-09-16 19:17:41.87642439 +0000 UTC m=+746.082380143" Sep 16 19:17:41.902014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:41.901935 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-cnxwn" podStartSLOduration=1.196463564 podStartE2EDuration="4.901920982s" totalImagesPullingTime="3.705457418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:41.899830363 +0000 UTC m=+746.105786115" watchObservedRunningTime="2026-09-16 19:17:41.901920982 +0000 UTC m=+746.107876731" Sep 16 19:17:52.866758 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:52.866728 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-bgs5l" Sep 16 19:17:52.868719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:52.868695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-cnxwn" Sep 16 19:17:53.788085 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.788048 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qd7tm"] Sep 16 19:17:53.791994 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.791971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.794902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.794884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 16 19:17:53.806720 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.806690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.806852 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.806735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkfr\" (UniqueName: \"kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.808171 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.808147 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qd7tm"] Sep 16 19:17:53.907831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.907797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.907831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.907839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkfr\" (UniqueName: \"kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.908225 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.908182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:53.917406 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:53.917383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkfr\" (UniqueName: \"kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr\") pod \"s3-init-qd7tm\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " pod="kserve/s3-init-qd7tm" Sep 16 19:17:54.101460 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:54.101376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qd7tm" Sep 16 19:17:54.376095 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:54.376070 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qd7tm"] Sep 16 19:17:54.378477 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:17:54.378451 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a61edf7_9fe0_438e_a59a_47bc6fe3ce64.slice/crio-688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654 WatchSource:0}: Error finding container 688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654: Status 404 returned error can't find the container with id 688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654 Sep 16 19:17:54.933636 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:54.933594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerStarted","Data":"688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654"} Sep 16 19:17:58.961352 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:17:58.961317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerStarted","Data":"342c26ab0922acb60471c0be989ce028100cd6ba2d520bee189d5fdd6005e9fa"} Sep 16 19:18:00.973721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:18:00.973685 2565 generic.go:356] "Generic (PLEG): container finished" podID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerID="342c26ab0922acb60471c0be989ce028100cd6ba2d520bee189d5fdd6005e9fa" exitCode=0 Sep 16 19:18:00.974116 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:18:00.973757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerDied","Data":"342c26ab0922acb60471c0be989ce028100cd6ba2d520bee189d5fdd6005e9fa"} Sep 16 19:18:07.016429 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:18:07.016395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerStarted","Data":"3d5283985943ed978a764dcc10512b770d7c1e78aa959364b61dde5ed8a3704a"} Sep 16 19:18:07.044022 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:18:07.043972 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qd7tm" podStartSLOduration=2.984511075 podStartE2EDuration="14.043955632s" totalImagesPullingTime="9.239466969s" totalInitContainerRuntime="1.819977588s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:07.042471774 +0000 UTC m=+771.248427531" watchObservedRunningTime="2026-09-16 19:18:07.043955632 +0000 UTC m=+771.249911384" Sep 16 19:19:12.375507 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:12.375475 2565 generic.go:356] "Generic (PLEG): container finished" podID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerID="3d5283985943ed978a764dcc10512b770d7c1e78aa959364b61dde5ed8a3704a" exitCode=0 Sep 16 19:19:12.375940 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:12.375545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerDied","Data":"3d5283985943ed978a764dcc10512b770d7c1e78aa959364b61dde5ed8a3704a"} Sep 16 19:19:13.520731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.520707 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qd7tm" Sep 16 19:19:13.618722 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.618690 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache\") pod \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " Sep 16 19:19:13.618884 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.618823 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgkfr\" (UniqueName: \"kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr\") pod \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\" (UID: \"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64\") " Sep 16 19:19:13.621037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.621016 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr" (OuterVolumeSpecName: "kube-api-access-kgkfr") pod "8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" (UID: "8a61edf7-9fe0-438e-a59a-47bc6fe3ce64"). InnerVolumeSpecName "kube-api-access-kgkfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:19:13.641462 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.641406 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" (UID: "8a61edf7-9fe0-438e-a59a-47bc6fe3ce64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:19:13.720017 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.719983 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkfr\" (UniqueName: \"kubernetes.io/projected/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-kube-api-access-kgkfr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:19:13.720017 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:13.720014 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a61edf7-9fe0-438e-a59a-47bc6fe3ce64-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:19:14.389862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:14.389836 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qd7tm" Sep 16 19:19:14.390036 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:14.389865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qd7tm" event={"ID":"8a61edf7-9fe0-438e-a59a-47bc6fe3ce64","Type":"ContainerDied","Data":"688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654"} Sep 16 19:19:14.390036 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:14.389899 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="688f3a0cdf86303bc6a82049c6fd0c8bded1d94f6beb489dc85017b4c4c23654" Sep 16 19:19:47.011058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011021 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:19:47.011544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011441 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerName="download-hf-model" Sep 16 19:19:47.011544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011454 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerName="download-hf-model" Sep 16 19:19:47.011544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011467 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerName="s3-init" Sep 16 19:19:47.011544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011473 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerName="s3-init" Sep 16 19:19:47.011544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.011529 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a61edf7-9fe0-438e-a59a-47bc6fe3ce64" containerName="s3-init" Sep 16 19:19:47.014778 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.014760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.021514 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021216 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap" Sep 16 19:19:47.021514 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021273 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rz87t\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rz87t\"" type="*v1.Secret" Sep 16 19:19:47.021514 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021310 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" type="*v1.Secret" Sep 16 19:19:47.021741 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021599 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\"" type="*v1.Secret" Sep 16 19:19:47.021806 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021765 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-gts24\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-gts24\"" type="*v1.Secret" Sep 16 19:19:47.021868 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021849 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 16 19:19:47.021919 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021854 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 16 19:19:47.021919 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.021857 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="6b340b0e-f980-4809-80b2-20963987422f" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.050334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.050277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:19:47.116599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clr74\" (UniqueName: \"kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116848 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.116978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.116899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217538 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217538 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.217791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clr74\" (UniqueName: \"kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.218101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.218101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.218101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.218101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.217981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.220050 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.220031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:47.419797 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.419719 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:19:47.423398 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.423377 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:47.431130 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:47.431105 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-9wrfr\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-63a03ef4\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-9wrfr\"" type="*v1.Secret" Sep 16 19:19:47.518300 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.518249 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:19:47.520255 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.520229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:47.520444 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.520427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gj4t\" (UniqueName: \"kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:47.621125 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.621083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:47.621376 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.621192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gj4t\" (UniqueName: \"kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:47.907206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.907171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:19:47.908594 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:47.908570 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:48.120525 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.120494 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-gts24\"" Sep 16 19:19:48.147039 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.147006 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\"" Sep 16 19:19:48.185785 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.185687 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\"" Sep 16 19:19:48.218668 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:48.218637 2565 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Sep 16 19:19:48.218836 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:19:48.218725 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs podName:6b340b0e-f980-4809-80b2-20963987422f nodeName:}" failed. No retries permitted until 2026-09-16 19:19:48.71870656 +0000 UTC m=+872.924662290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" (UID: "6b340b0e-f980-4809-80b2-20963987422f") : failed to sync secret cache: timed out waiting for the condition Sep 16 19:19:48.243095 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.243064 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\"" Sep 16 19:19:48.246477 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.246453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clr74\" (UniqueName: \"kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:48.251927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.251898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gj4t\" (UniqueName: \"kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:48.333381 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.333347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" Sep 16 19:19:48.344504 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.344477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:48.596378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.596348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rz87t\"" Sep 16 19:19:48.731092 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.731056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:48.733559 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.733537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:48.752145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.752119 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-9wrfr\"" Sep 16 19:19:48.754189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.754173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:19:48.824775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:48.824711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:19:49.049176 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:19:49.049149 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4ec1ff_c81e_49af_bad5_10a8603cd4bc.slice/crio-e2202a57e8dfd43ec1f65066f5a03eefbeaca396c4125bbe64f888bf75dbb5d5 WatchSource:0}: Error finding container e2202a57e8dfd43ec1f65066f5a03eefbeaca396c4125bbe64f888bf75dbb5d5: Status 404 returned error can't find the container with id e2202a57e8dfd43ec1f65066f5a03eefbeaca396c4125bbe64f888bf75dbb5d5 Sep 16 19:19:49.049561 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:49.049539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:19:49.132273 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:49.132247 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:19:49.134250 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:19:49.134224 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b340b0e_f980_4809_80b2_20963987422f.slice/crio-796d7ed9b0c629655b715ec29f936d53c7f7c2b094d027589734a83bccb1d08e WatchSource:0}: Error finding container 796d7ed9b0c629655b715ec29f936d53c7f7c2b094d027589734a83bccb1d08e: Status 404 returned error can't find the container with id 796d7ed9b0c629655b715ec29f936d53c7f7c2b094d027589734a83bccb1d08e Sep 16 19:19:49.595426 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:49.595383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerStarted","Data":"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884"} Sep 16 19:19:49.595633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:49.595459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerStarted","Data":"796d7ed9b0c629655b715ec29f936d53c7f7c2b094d027589734a83bccb1d08e"} Sep 16 19:19:49.596734 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:49.596710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" event={"ID":"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc","Type":"ContainerStarted","Data":"e2202a57e8dfd43ec1f65066f5a03eefbeaca396c4125bbe64f888bf75dbb5d5"} Sep 16 19:19:52.622251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:52.622217 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b340b0e-f980-4809-80b2-20963987422f" containerID="4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884" exitCode=0 Sep 16 19:19:52.622784 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:52.622323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerDied","Data":"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884"} Sep 16 19:19:52.623960 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:52.623932 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" event={"ID":"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc","Type":"ContainerStarted","Data":"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd"} Sep 16 19:19:52.624080 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:19:52.624064 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:20:17.842231 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:17.842117 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podStartSLOduration=28.230432946 podStartE2EDuration="30.842094325s" totalImagesPullingTime="2.611661379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:52.693413 +0000 UTC m=+876.899368752" watchObservedRunningTime="2026-09-16 19:20:17.842094325 +0000 UTC m=+902.048050084" Sep 16 19:20:17.842838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:17.842274 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:20:18.051433 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.051390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:20:18.051623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.051514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.058924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.058786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\"" Sep 16 19:20:18.058924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.058804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\"" Sep 16 19:20:18.058924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.058820 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-gk6z8\"" Sep 16 19:20:18.058924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.058830 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\"" Sep 16 19:20:18.139077 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.139077 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.139325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.139325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.139325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9hn\" (UniqueName: \"kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.139478 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.139341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9hn\" (UniqueName: \"kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240997 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.240997 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.240973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.241258 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.241230 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.241496 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.241464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.243628 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.243609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.243985 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.243962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.253006 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.252981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9hn\" (UniqueName: \"kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn\") pod \"fc-smoke-test-kserve-7f4687897-rdv6s\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:18.364046 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:18.364004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:20.717252 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:20.717219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:20:20.774250 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:20:20.774217 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3aa435_2def_4247_8740_8732cdaa524e.slice/crio-da7a500aab06843570662aca6169b4330995466a30e98e14d42002fc08e94fce WatchSource:0}: Error finding container da7a500aab06843570662aca6169b4330995466a30e98e14d42002fc08e94fce: Status 404 returned error can't find the container with id da7a500aab06843570662aca6169b4330995466a30e98e14d42002fc08e94fce Sep 16 19:20:20.835845 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:20.835801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" event={"ID":"eb3aa435-2def-4247-8740-8732cdaa524e","Type":"ContainerStarted","Data":"da7a500aab06843570662aca6169b4330995466a30e98e14d42002fc08e94fce"} Sep 16 19:20:21.852506 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:21.852449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerStarted","Data":"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89"} Sep 16 19:20:21.885098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:21.885030 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=4.983345189 podStartE2EDuration="35.885008185s" totalImagesPullingTime="28.265506762s" totalInitContainerRuntime="2.636156234s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:21.883093274 +0000 UTC m=+906.089049031" watchObservedRunningTime="2026-09-16 19:20:21.885008185 +0000 UTC m=+906.090963940" Sep 16 19:20:22.825199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:22.825161 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:20:23.635753 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:23.635660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:20:23.868805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:23.868771 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" event={"ID":"eb3aa435-2def-4247-8740-8732cdaa524e","Type":"ContainerStarted","Data":"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f"} Sep 16 19:20:23.933421 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:23.933317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" podStartSLOduration=4.493002715 podStartE2EDuration="6.933296941s" totalImagesPullingTime="2.440294226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:23.931637977 +0000 UTC m=+908.137593730" watchObservedRunningTime="2026-09-16 19:20:23.933296941 +0000 UTC m=+908.139252684" Sep 16 19:20:24.364486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:24.364454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:28.364611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:28.364570 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:28.378540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:28.378512 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:28.825762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:28.825721 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:20:28.827219 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:28.827188 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:20:28.914804 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:28.914773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:20:38.825819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:38.825773 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:20:48.826071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:48.826025 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:20:58.825690 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:20:58.825643 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:08.826019 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:08.825975 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:09.231702 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.231670 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:21:09.241567 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:21:09.241532 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:21:09.241567 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:21:09.241567 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:21:09.241567 ip-10-0-128-148 kubenswrapper[2565]: > containerID="78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" cmd=["/bin/sleep","15"] Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.241591 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" message="" err=< Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:21:09.241606 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" containerName="main" containerID="cri-o://78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" Sep 16 19:21:09.241820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.241645 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" containerName="main" containerID="cri-o://78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" gracePeriod=60 Sep 16 19:21:09.607254 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.607225 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:21:09.694384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694437 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694536 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694550 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj9hn\" (UniqueName: \"kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn\") pod \"eb3aa435-2def-4247-8740-8732cdaa524e\" (UID: \"eb3aa435-2def-4247-8740-8732cdaa524e\") " Sep 16 19:21:09.694691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:21:09.695029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home" (OuterVolumeSpecName: "home") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:21:09.695029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694848 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:21:09.695029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694928 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:09.695029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694948 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:09.695029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.694961 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:09.696850 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.696821 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:21:09.696956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.696845 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm" (OuterVolumeSpecName: "dshm") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:21:09.696956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.696908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn" (OuterVolumeSpecName: "kube-api-access-pj9hn") pod "eb3aa435-2def-4247-8740-8732cdaa524e" (UID: "eb3aa435-2def-4247-8740-8732cdaa524e"). InnerVolumeSpecName "kube-api-access-pj9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:21:09.795766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.795678 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3aa435-2def-4247-8740-8732cdaa524e-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:09.795766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.795710 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9hn\" (UniqueName: \"kubernetes.io/projected/eb3aa435-2def-4247-8740-8732cdaa524e-kube-api-access-pj9hn\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:09.795766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:09.795729 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3aa435-2def-4247-8740-8732cdaa524e-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:21:10.169209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.169115 2565 generic.go:356] "Generic (PLEG): container finished" podID="eb3aa435-2def-4247-8740-8732cdaa524e" containerID="78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" exitCode=0 Sep 16 19:21:10.169209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.169200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" event={"ID":"eb3aa435-2def-4247-8740-8732cdaa524e","Type":"ContainerDied","Data":"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f"} Sep 16 19:21:10.169721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.169223 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" Sep 16 19:21:10.169721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.169253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s" event={"ID":"eb3aa435-2def-4247-8740-8732cdaa524e","Type":"ContainerDied","Data":"da7a500aab06843570662aca6169b4330995466a30e98e14d42002fc08e94fce"} Sep 16 19:21:10.169721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.169314 2565 scope.go:117] "RemoveContainer" containerID="78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" Sep 16 19:21:10.179846 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.179828 2565 scope.go:117] "RemoveContainer" containerID="78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" Sep 16 19:21:10.180199 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:21:10.180173 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f\": container with ID starting with 78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f not found: ID does not exist" containerID="78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f" Sep 16 19:21:10.180356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.180204 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f"} err="failed to get container status \"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f\": rpc error: code = NotFound desc = could not find container \"78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f\": container with ID starting with 78499077af5357cb0e855ff907f1abdce24e564d4984b7ea532c55526475576f not found: ID does not exist" Sep 16 19:21:10.255561 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.255527 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:21:10.255967 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:10.255946 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rdv6s"] Sep 16 19:21:12.253596 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:12.253560 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" path="/var/lib/kubelet/pods/eb3aa435-2def-4247-8740-8732cdaa524e/volumes" Sep 16 19:21:18.825582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:18.825541 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:28.825299 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:28.825232 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:38.826085 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:38.825987 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:48.825573 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:48.825532 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused" Sep 16 19:21:58.835165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:58.835134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:21:58.842499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:21:58.842475 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:22:04.002996 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.002960 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:04.003443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.003428 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" containerName="main" Sep 16 19:22:04.003491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.003445 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" containerName="main" Sep 16 19:22:04.003526 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.003508 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3aa435-2def-4247-8740-8732cdaa524e" containerName="main" Sep 16 19:22:04.008976 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.008956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.014139 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.014105 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-dcns9\"" Sep 16 19:22:04.015026 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.015004 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\"" Sep 16 19:22:04.015026 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.015016 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\"" Sep 16 19:22:04.015199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.015006 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\"" Sep 16 19:22:04.098510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.098706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.098706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098598 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.098706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098657 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.098706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.098706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.098695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ftwx\" (UniqueName: \"kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.130965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.130924 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:04.200329 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200694 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ftwx\" (UniqueName: \"kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200694 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.200910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.200841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.202838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.202808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.203363 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.203343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.220163 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.220132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ftwx\" (UniqueName: \"kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx\") pod \"fc-concurrency-test-kserve-b5d44467c-6tm5j\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.319598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.319513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:04.615138 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:04.615110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:04.617518 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:22:04.617489 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4db0a07_b3ed_42fe_a692_4d0c03f454d1.slice/crio-520c5ad74bb44a4ffe9cded378cba2cb392aba0e81fa873b12e2d8af6be26922 WatchSource:0}: Error finding container 520c5ad74bb44a4ffe9cded378cba2cb392aba0e81fa873b12e2d8af6be26922: Status 404 returned error can't find the container with id 520c5ad74bb44a4ffe9cded378cba2cb392aba0e81fa873b12e2d8af6be26922 Sep 16 19:22:05.515516 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:05.515462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" event={"ID":"d4db0a07-b3ed-42fe-a692-4d0c03f454d1","Type":"ContainerStarted","Data":"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312"} Sep 16 19:22:05.516097 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:05.515557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" event={"ID":"d4db0a07-b3ed-42fe-a692-4d0c03f454d1","Type":"ContainerStarted","Data":"520c5ad74bb44a4ffe9cded378cba2cb392aba0e81fa873b12e2d8af6be26922"} Sep 16 19:22:05.549169 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:05.549107 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" podStartSLOduration=2.54909109 podStartE2EDuration="2.54909109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:05.5472749 +0000 UTC m=+1009.753230652" watchObservedRunningTime="2026-09-16 19:22:05.54909109 +0000 UTC m=+1009.755046842" Sep 16 19:22:06.319661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:06.319623 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:07.731125 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:07.731092 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:22:07.747373 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:07.747342 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:22:13.634761 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:13.634721 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:22:14.319880 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:14.319838 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:14.332382 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:14.332356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:14.582156 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:14.582072 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:22.751445 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:22.751378 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" containerID="cri-o://71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd" gracePeriod=45 Sep 16 19:22:22.769338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:22.769296 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" containerID="cri-o://f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89" gracePeriod=45 Sep 16 19:22:23.531327 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.531300 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:22:23.633058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.632963 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b340b0e-f980-4809-80b2-20963987422f" containerID="f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89" exitCode=0 Sep 16 19:22:23.633058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.633029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerDied","Data":"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89"} Sep 16 19:22:23.633058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.633033 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 16 19:22:23.633320 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.633073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6b340b0e-f980-4809-80b2-20963987422f","Type":"ContainerDied","Data":"796d7ed9b0c629655b715ec29f936d53c7f7c2b094d027589734a83bccb1d08e"} Sep 16 19:22:23.633320 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.633094 2565 scope.go:117] "RemoveContainer" containerID="f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89" Sep 16 19:22:23.634873 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.634837 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:22:23.642373 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.642349 2565 scope.go:117] "RemoveContainer" containerID="4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884" Sep 16 19:22:23.650736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.650719 2565 scope.go:117] "RemoveContainer" containerID="f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89" Sep 16 19:22:23.651008 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:23.650987 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89\": container with ID starting with f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89 not found: ID does not exist" containerID="f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89" Sep 16 19:22:23.651078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.651014 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89"} err="failed to get container status \"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89\": rpc error: code = NotFound desc = could not find container \"f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89\": container with ID starting with f1dbf5fc0e623de8b7130054b8a413b937e832e1fa76600bb09b97f997dc8f89 not found: ID does not exist" Sep 16 19:22:23.651078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.651031 2565 scope.go:117] "RemoveContainer" containerID="4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884" Sep 16 19:22:23.651266 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:23.651249 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884\": container with ID starting with 4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884 not found: ID does not exist" containerID="4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884" Sep 16 19:22:23.651356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.651273 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884"} err="failed to get container status \"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884\": rpc error: code = NotFound desc = could not find container \"4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884\": container with ID starting with 4082f1ee124d9481b4e5c8b5d4ef40458580a6eb19fcbc877fedcd81564d0884 not found: ID does not exist" Sep 16 19:22:23.687051 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687018 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687051 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687051 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687121 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687157 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687178 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clr74\" (UniqueName: \"kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687248 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687205 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687591 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir\") pod \"6b340b0e-f980-4809-80b2-20963987422f\" (UID: \"6b340b0e-f980-4809-80b2-20963987422f\") " Sep 16 19:22:23.687730 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687661 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:23.687861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687745 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:22:23.687927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.687904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home" (OuterVolumeSpecName: "home") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:23.689855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.689832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:22:23.689942 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.689889 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm" (OuterVolumeSpecName: "dshm") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:23.690075 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.690055 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74" (OuterVolumeSpecName: "kube-api-access-clr74") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "kube-api-access-clr74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:22:23.700494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.700469 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:23.709330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.709310 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b340b0e-f980-4809-80b2-20963987422f" (UID: "6b340b0e-f980-4809-80b2-20963987422f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:23.788701 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788663 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.788701 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788692 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.788701 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788702 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b340b0e-f980-4809-80b2-20963987422f-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.788701 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788712 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clr74\" (UniqueName: \"kubernetes.io/projected/6b340b0e-f980-4809-80b2-20963987422f-kube-api-access-clr74\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.789165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788720 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.789165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788728 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.789165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788735 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b340b0e-f980-4809-80b2-20963987422f-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.789165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.788743 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b340b0e-f980-4809-80b2-20963987422f-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:23.962456 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.962419 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:22:23.966476 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:23.966449 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 16 19:22:24.253885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:24.253855 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b340b0e-f980-4809-80b2-20963987422f" path="/var/lib/kubelet/pods/6b340b0e-f980-4809-80b2-20963987422f/volumes" Sep 16 19:22:33.635381 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:33.635344 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:22:33.635848 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:33.635459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:22:43.635348 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:43.635267 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:22:43.780097 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:43.780060 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:43.789878 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:43.789826 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:22:43.789878 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:22:43.789878 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:22:43.789878 ip-10-0-128-148 kubenswrapper[2565]: > containerID="33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" cmd=["/bin/sleep","15"] Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:43.789894 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" message="" err=< Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:43.789907 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerName="main" containerID="cri-o://33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" Sep 16 19:22:43.790158 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:43.789949 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerName="main" containerID="cri-o://33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" gracePeriod=60 Sep 16 19:22:44.148411 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.148345 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:44.173900 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.173866 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.173910 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.173937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ftwx\" (UniqueName: \"kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.173975 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174011 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174087 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache\") pod \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\" (UID: \"d4db0a07-b3ed-42fe-a692-4d0c03f454d1\") " Sep 16 19:22:44.174398 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174182 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:44.174398 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174276 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home" (OuterVolumeSpecName: "home") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:44.174505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:44.174505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174472 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.174505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.174499 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.176706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.176623 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx" (OuterVolumeSpecName: "kube-api-access-9ftwx") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "kube-api-access-9ftwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:22:44.176706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.176667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm" (OuterVolumeSpecName: "dshm") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:22:44.177012 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.176997 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4db0a07-b3ed-42fe-a692-4d0c03f454d1" (UID: "d4db0a07-b3ed-42fe-a692-4d0c03f454d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:22:44.275560 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.275531 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.275560 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.275557 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ftwx\" (UniqueName: \"kubernetes.io/projected/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-kube-api-access-9ftwx\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.275745 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.275566 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.275745 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.275577 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4db0a07-b3ed-42fe-a692-4d0c03f454d1-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:44.756193 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.756155 2565 generic.go:356] "Generic (PLEG): container finished" podID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerID="33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" exitCode=0 Sep 16 19:22:44.756639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.756223 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" Sep 16 19:22:44.756639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.756237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" event={"ID":"d4db0a07-b3ed-42fe-a692-4d0c03f454d1","Type":"ContainerDied","Data":"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312"} Sep 16 19:22:44.756639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.756306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j" event={"ID":"d4db0a07-b3ed-42fe-a692-4d0c03f454d1","Type":"ContainerDied","Data":"520c5ad74bb44a4ffe9cded378cba2cb392aba0e81fa873b12e2d8af6be26922"} Sep 16 19:22:44.756639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.756330 2565 scope.go:117] "RemoveContainer" containerID="33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" Sep 16 19:22:44.765922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.765901 2565 scope.go:117] "RemoveContainer" containerID="33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" Sep 16 19:22:44.766241 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:44.766222 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312\": container with ID starting with 33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312 not found: ID does not exist" containerID="33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312" Sep 16 19:22:44.766316 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.766251 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312"} err="failed to get container status \"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312\": rpc error: code = NotFound desc = could not find container \"33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312\": container with ID starting with 33a83fb8b086748528acdae19dc797e1c2235254eb6892ddaee49ce588b66312 not found: ID does not exist" Sep 16 19:22:44.778375 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.778341 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:44.783073 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:44.783052 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-6tm5j"] Sep 16 19:22:46.253984 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:46.253953 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" path="/var/lib/kubelet/pods/d4db0a07-b3ed-42fe-a692-4d0c03f454d1/volumes" Sep 16 19:22:53.003440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.003414 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:22:53.047894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.047820 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs\") pod \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " Sep 16 19:22:53.047894 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.047873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gj4t\" (UniqueName: \"kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t\") pod \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\" (UID: \"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc\") " Sep 16 19:22:53.050106 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.050076 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t" (OuterVolumeSpecName: "kube-api-access-4gj4t") pod "4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" (UID: "4b4ec1ff-c81e-49af-bad5-10a8603cd4bc"). InnerVolumeSpecName "kube-api-access-4gj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:22:53.050199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.050171 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" (UID: "4b4ec1ff-c81e-49af-bad5-10a8603cd4bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:22:53.149333 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.149271 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:53.149333 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.149330 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gj4t\" (UniqueName: \"kubernetes.io/projected/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc-kube-api-access-4gj4t\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:22:53.811243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.811212 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerID="71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd" exitCode=0 Sep 16 19:22:53.811527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.811257 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" Sep 16 19:22:53.811527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.811309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" event={"ID":"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc","Type":"ContainerDied","Data":"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd"} Sep 16 19:22:53.811527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.811366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn" event={"ID":"4b4ec1ff-c81e-49af-bad5-10a8603cd4bc","Type":"ContainerDied","Data":"e2202a57e8dfd43ec1f65066f5a03eefbeaca396c4125bbe64f888bf75dbb5d5"} Sep 16 19:22:53.811527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.811388 2565 scope.go:117] "RemoveContainer" containerID="71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd" Sep 16 19:22:53.820985 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.820969 2565 scope.go:117] "RemoveContainer" containerID="71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd" Sep 16 19:22:53.821352 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:22:53.821318 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd\": container with ID starting with 71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd not found: ID does not exist" containerID="71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd" Sep 16 19:22:53.821424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.821361 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd"} err="failed to get container status \"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd\": rpc error: code = NotFound desc = could not find container \"71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd\": container with ID starting with 71f503e639b247cf4e10f137ec094868d959c21afcdc9ba0ed478a8bf88ca2cd not found: ID does not exist" Sep 16 19:22:53.845710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.845679 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:22:53.861066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:53.861033 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schevhfvn"] Sep 16 19:22:54.253708 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:54.253675 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" path="/var/lib/kubelet/pods/4b4ec1ff-c81e-49af-bad5-10a8603cd4bc/volumes" Sep 16 19:22:58.989776 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.989739 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59cf576fd8-plg8d"] Sep 16 19:22:58.990176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990140 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerName="main" Sep 16 19:22:58.990176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990153 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerName="main" Sep 16 19:22:58.990176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990167 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" Sep 16 19:22:58.990176 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990173 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990182 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="storage-initializer" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990188 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="storage-initializer" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990200 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990205 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990293 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4ec1ff-c81e-49af-bad5-10a8603cd4bc" containerName="main" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990304 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b340b0e-f980-4809-80b2-20963987422f" containerName="main" Sep 16 19:22:58.990338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.990318 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4db0a07-b3ed-42fe-a692-4d0c03f454d1" containerName="main" Sep 16 19:22:58.994666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:58.994648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.027840 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.027800 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cf576fd8-plg8d"] Sep 16 19:22:59.103518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-oauth-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-trusted-ca-bundle\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-oauth-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4wx\" (UniqueName: \"kubernetes.io/projected/df06f925-fb23-4853-9e51-aa586209fd51-kube-api-access-9c4wx\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-service-ca\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.103822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.103756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-console-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.204734 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-oauth-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.204734 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4wx\" (UniqueName: \"kubernetes.io/projected/df06f925-fb23-4853-9e51-aa586209fd51-kube-api-access-9c4wx\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-service-ca\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-console-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-oauth-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.204856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205007 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.205001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-trusted-ca-bundle\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205581 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.205557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-oauth-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205804 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.205781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-console-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205848 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.205802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-service-ca\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.205848 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.205787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df06f925-fb23-4853-9e51-aa586209fd51-trusted-ca-bundle\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.207570 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.207545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-oauth-config\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.207655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.207603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df06f925-fb23-4853-9e51-aa586209fd51-console-serving-cert\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.218086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.218063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4wx\" (UniqueName: \"kubernetes.io/projected/df06f925-fb23-4853-9e51-aa586209fd51-kube-api-access-9c4wx\") pod \"console-59cf576fd8-plg8d\" (UID: \"df06f925-fb23-4853-9e51-aa586209fd51\") " pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.303994 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.303912 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:22:59.610071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.609996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cf576fd8-plg8d"] Sep 16 19:22:59.614820 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:22:59.614789 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf06f925_fb23_4853_9e51_aa586209fd51.slice/crio-2ada842ed3a02adf96fab40996dea2529d5178df61b98069f25a9627217fe3f6 WatchSource:0}: Error finding container 2ada842ed3a02adf96fab40996dea2529d5178df61b98069f25a9627217fe3f6: Status 404 returned error can't find the container with id 2ada842ed3a02adf96fab40996dea2529d5178df61b98069f25a9627217fe3f6 Sep 16 19:22:59.849396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.849357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cf576fd8-plg8d" event={"ID":"df06f925-fb23-4853-9e51-aa586209fd51","Type":"ContainerStarted","Data":"9e7a273a9e8313cff3adf77c11846167287556fe2869d39441d9e4f8295551b9"} Sep 16 19:22:59.849396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.849405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cf576fd8-plg8d" event={"ID":"df06f925-fb23-4853-9e51-aa586209fd51","Type":"ContainerStarted","Data":"2ada842ed3a02adf96fab40996dea2529d5178df61b98069f25a9627217fe3f6"} Sep 16 19:22:59.896426 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:22:59.896317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59cf576fd8-plg8d" podStartSLOduration=1.8962967590000002 podStartE2EDuration="1.896296759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:59.89605902 +0000 UTC m=+1064.102014773" watchObservedRunningTime="2026-09-16 19:22:59.896296759 +0000 UTC m=+1064.102252504" Sep 16 19:23:09.304685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:09.304595 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:23:09.304685 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:09.304654 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:23:09.309714 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:09.309693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:23:09.913787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:09.913760 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59cf576fd8-plg8d" Sep 16 19:23:10.024583 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:10.024553 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:23:13.414837 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.414801 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:23:13.418259 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.418242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.422335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.422313 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\"" Sep 16 19:23:13.422601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.422578 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\"" Sep 16 19:23:13.423483 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.423464 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-bzbn7\"" Sep 16 19:23:13.423483 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.423473 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" Sep 16 19:23:13.527409 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.527373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:23:13.542965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.542930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.543190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.542983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.543190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.543005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.543190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.543047 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcxr\" (UniqueName: \"kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.543190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.543107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.543190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.543164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644069 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcxr\" (UniqueName: \"kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644236 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644598 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644573 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.644663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.644635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.646662 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.646638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.646803 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.646786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.655619 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.655591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcxr\" (UniqueName: \"kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr\") pod \"scheduler-inline-config-test-kserve-8655bc757-6ftv9\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:13.728696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:13.728669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:14.040890 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:14.040833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:23:14.043561 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:23:14.043532 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode614fa46_3d00_4520_8d12_78a968ff32c5.slice/crio-bd5559161ee05a88881cebdf963339f42645aa36093e724983619741cae2e4da WatchSource:0}: Error finding container bd5559161ee05a88881cebdf963339f42645aa36093e724983619741cae2e4da: Status 404 returned error can't find the container with id bd5559161ee05a88881cebdf963339f42645aa36093e724983619741cae2e4da Sep 16 19:23:14.940763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:14.940722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" event={"ID":"e614fa46-3d00-4520-8d12-78a968ff32c5","Type":"ContainerStarted","Data":"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba"} Sep 16 19:23:14.941204 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:14.940786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" event={"ID":"e614fa46-3d00-4520-8d12-78a968ff32c5","Type":"ContainerStarted","Data":"bd5559161ee05a88881cebdf963339f42645aa36093e724983619741cae2e4da"} Sep 16 19:23:14.994230 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:14.994164 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" podStartSLOduration=1.994139603 podStartE2EDuration="1.994139603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:14.991252335 +0000 UTC m=+1079.197208087" watchObservedRunningTime="2026-09-16 19:23:14.994139603 +0000 UTC m=+1079.200095388" Sep 16 19:23:15.729026 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:15.728992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:23.729769 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:23.729729 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:23.742838 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:23.742814 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:24.003335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:24.003235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:23:35.044398 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.044336 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f57d597bc-8rptz" podUID="7760cba9-3055-4c66-a344-478182cfe61c" containerName="console" containerID="cri-o://78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177" gracePeriod=15 Sep 16 19:23:35.301161 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.301099 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:23:35.335663 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335632 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.335841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335711 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99hmf\" (UniqueName: \"kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.335841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335770 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.335841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335796 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.335841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335832 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.336069 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335872 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.336069 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.335912 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config\") pod \"7760cba9-3055-4c66-a344-478182cfe61c\" (UID: \"7760cba9-3055-4c66-a344-478182cfe61c\") " Sep 16 19:23:35.336308 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.336240 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:23:35.336434 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.336323 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:23:35.336726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.336497 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config" (OuterVolumeSpecName: "console-config") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:23:35.336726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.336627 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca" (OuterVolumeSpecName: "service-ca") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:23:35.338259 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.338229 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf" (OuterVolumeSpecName: "kube-api-access-99hmf") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "kube-api-access-99hmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:23:35.338424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.338339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:23:35.338671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.338651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7760cba9-3055-4c66-a344-478182cfe61c" (UID: "7760cba9-3055-4c66-a344-478182cfe61c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:23:35.437243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437203 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-oauth-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437234 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-trusted-ca-bundle\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437248 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-oauth-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437259 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7760cba9-3055-4c66-a344-478182cfe61c-console-serving-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437272 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-console-config\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437302 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7760cba9-3055-4c66-a344-478182cfe61c-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:35.437494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:35.437312 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hmf\" (UniqueName: \"kubernetes.io/projected/7760cba9-3055-4c66-a344-478182cfe61c-kube-api-access-99hmf\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:36.063134 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.063100 2565 generic.go:356] "Generic (PLEG): container finished" podID="7760cba9-3055-4c66-a344-478182cfe61c" containerID="78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177" exitCode=0 Sep 16 19:23:36.063699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.063161 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f57d597bc-8rptz" Sep 16 19:23:36.063699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.063192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f57d597bc-8rptz" event={"ID":"7760cba9-3055-4c66-a344-478182cfe61c","Type":"ContainerDied","Data":"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177"} Sep 16 19:23:36.063699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.063245 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f57d597bc-8rptz" event={"ID":"7760cba9-3055-4c66-a344-478182cfe61c","Type":"ContainerDied","Data":"50c0aa5f19a378a1449d2941556e169e501f8270a3767fa85b921c24b0b167d0"} Sep 16 19:23:36.063699 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.063265 2565 scope.go:117] "RemoveContainer" containerID="78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177" Sep 16 19:23:36.074420 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.074404 2565 scope.go:117] "RemoveContainer" containerID="78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177" Sep 16 19:23:36.074712 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:36.074692 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177\": container with ID starting with 78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177 not found: ID does not exist" containerID="78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177" Sep 16 19:23:36.074781 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.074721 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177"} err="failed to get container status \"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177\": rpc error: code = NotFound desc = could not find container \"78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177\": container with ID starting with 78700c4d7b07e87e6c41263b6f1b06825d7fc8a5a1538d562e2dc477b5946177 not found: ID does not exist" Sep 16 19:23:36.109637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.109598 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:23:36.121560 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.121528 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f57d597bc-8rptz"] Sep 16 19:23:36.260432 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:36.260398 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7760cba9-3055-4c66-a344-478182cfe61c" path="/var/lib/kubelet/pods/7760cba9-3055-4c66-a344-478182cfe61c/volumes" Sep 16 19:23:39.075335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.075265 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:39.075919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.075898 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7760cba9-3055-4c66-a344-478182cfe61c" containerName="console" Sep 16 19:23:39.076004 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.075923 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7760cba9-3055-4c66-a344-478182cfe61c" containerName="console" Sep 16 19:23:39.076060 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.076040 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7760cba9-3055-4c66-a344-478182cfe61c" containerName="console" Sep 16 19:23:39.080635 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.080615 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.083519 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.083498 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\"" Sep 16 19:23:39.084142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.084126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Sep 16 19:23:39.084142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.084136 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-pfb5x\"" Sep 16 19:23:39.084269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.084136 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\"" Sep 16 19:23:39.101931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.101905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:39.173173 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.173387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.173387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx7q\" (UniqueName: \"kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.173387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.173387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.173387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.173354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274022 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.273966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx7q\" (UniqueName: \"kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.274639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.274580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.276425 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.276405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.276672 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.276654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.352778 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.352705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx7q\" (UniqueName: \"kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:39.391585 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:39.391553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:41.926739 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:41.926688 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:41.930991 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:23:41.930964 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2de0c55_2e52_47b0_81dd_c29a13763351.slice/crio-ab9d18667e14bb75a0a3f07dc90e253278dbf99581a1dac1bdb3002d6b861b9a WatchSource:0}: Error finding container ab9d18667e14bb75a0a3f07dc90e253278dbf99581a1dac1bdb3002d6b861b9a: Status 404 returned error can't find the container with id ab9d18667e14bb75a0a3f07dc90e253278dbf99581a1dac1bdb3002d6b861b9a Sep 16 19:23:42.104427 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:42.104392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" event={"ID":"c2de0c55-2e52-47b0-81dd-c29a13763351","Type":"ContainerStarted","Data":"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e"} Sep 16 19:23:42.104586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:42.104452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" event={"ID":"c2de0c55-2e52-47b0-81dd-c29a13763351","Type":"ContainerStarted","Data":"ab9d18667e14bb75a0a3f07dc90e253278dbf99581a1dac1bdb3002d6b861b9a"} Sep 16 19:23:42.138043 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:42.137990 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" podStartSLOduration=3.137966333 podStartE2EDuration="3.137966333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:42.136190927 +0000 UTC m=+1106.342146679" watchObservedRunningTime="2026-09-16 19:23:42.137966333 +0000 UTC m=+1106.343922085" Sep 16 19:23:42.391811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:42.391776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:46.573513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.573476 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:46.585826 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:46.585783 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:23:46.585826 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:46.585826 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:46.585826 ip-10-0-128-148 kubenswrapper[2565]: > containerID="66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" cmd=["/bin/sleep","15"] Sep 16 19:23:46.586203 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.586178 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" message="" err=< Sep 16 19:23:46.586203 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:46.586203 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:46.586203 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:23:46.586458 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:46.586331 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:23:46.586458 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:46.586458 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:46.586458 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerName="main" containerID="cri-o://66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" Sep 16 19:23:46.586458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.586386 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerName="main" containerID="cri-o://66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" gracePeriod=60 Sep 16 19:23:46.924208 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.924173 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:46.957658 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.957861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.957861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957719 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxx7q\" (UniqueName: \"kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.957861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.957861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.957861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache\") pod \"c2de0c55-2e52-47b0-81dd-c29a13763351\" (UID: \"c2de0c55-2e52-47b0-81dd-c29a13763351\") " Sep 16 19:23:46.958150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.957916 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home" (OuterVolumeSpecName: "home") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:46.958150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.958068 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:46.958150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.958143 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:46.958321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.958159 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:46.958321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.958227 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:46.961618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.960843 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm" (OuterVolumeSpecName: "dshm") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:46.961618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.961427 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:23:46.961618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:46.961563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q" (OuterVolumeSpecName: "kube-api-access-qxx7q") pod "c2de0c55-2e52-47b0-81dd-c29a13763351" (UID: "c2de0c55-2e52-47b0-81dd-c29a13763351"). InnerVolumeSpecName "kube-api-access-qxx7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:23:47.058931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.058899 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:47.058931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.058928 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxx7q\" (UniqueName: \"kubernetes.io/projected/c2de0c55-2e52-47b0-81dd-c29a13763351-kube-api-access-qxx7q\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:47.058931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.058938 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2de0c55-2e52-47b0-81dd-c29a13763351-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:47.059170 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.058949 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2de0c55-2e52-47b0-81dd-c29a13763351-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:47.135227 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.135143 2565 generic.go:356] "Generic (PLEG): container finished" podID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerID="66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" exitCode=0 Sep 16 19:23:47.135428 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.135225 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" Sep 16 19:23:47.135428 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.135236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" event={"ID":"c2de0c55-2e52-47b0-81dd-c29a13763351","Type":"ContainerDied","Data":"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e"} Sep 16 19:23:47.135428 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.135312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27" event={"ID":"c2de0c55-2e52-47b0-81dd-c29a13763351","Type":"ContainerDied","Data":"ab9d18667e14bb75a0a3f07dc90e253278dbf99581a1dac1bdb3002d6b861b9a"} Sep 16 19:23:47.135428 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.135338 2565 scope.go:117] "RemoveContainer" containerID="66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" Sep 16 19:23:47.148511 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.148486 2565 scope.go:117] "RemoveContainer" containerID="66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" Sep 16 19:23:47.148842 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:47.148823 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e\": container with ID starting with 66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e not found: ID does not exist" containerID="66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e" Sep 16 19:23:47.148907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.148853 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e"} err="failed to get container status \"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e\": rpc error: code = NotFound desc = could not find container \"66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e\": container with ID starting with 66c6d20cb567b57ce351a96b69f65ade53d0efc58963ba77b1a69aa38351f06e not found: ID does not exist" Sep 16 19:23:47.169308 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.168540 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:47.192174 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:47.192133 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-rxs27"] Sep 16 19:23:48.254107 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:48.254075 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" path="/var/lib/kubelet/pods/c2de0c55-2e52-47b0-81dd-c29a13763351/volumes" Sep 16 19:23:52.839102 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.839065 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:23:52.839544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.839528 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerName="main" Sep 16 19:23:52.839544 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.839546 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerName="main" Sep 16 19:23:52.839618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.839608 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2de0c55-2e52-47b0-81dd-c29a13763351" containerName="main" Sep 16 19:23:52.844668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.844645 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.851937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.851897 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\"" Sep 16 19:23:52.852616 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.852595 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Sep 16 19:23:52.852704 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.852650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\"" Sep 16 19:23:52.853968 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.853952 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-9bghm\"" Sep 16 19:23:52.860696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.860671 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:23:52.915618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.915825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.915825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.915825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.915825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:52.916001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:52.915833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snjm\" (UniqueName: \"kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017256 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snjm\" (UniqueName: \"kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017796 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017863 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.017901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.017856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.019766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.019746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.020009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.019993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.027474 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.027452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snjm\" (UniqueName: \"kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.156624 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.156535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:53.465367 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:53.465333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:23:53.465490 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:23:53.465379 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5ac0ed_a593_4be6_a662_7bd2dc35d39b.slice/crio-be9ccc983877625182a1d7b0d806afdeb3d10049e7429e3ece7e225a7b50ed99 WatchSource:0}: Error finding container be9ccc983877625182a1d7b0d806afdeb3d10049e7429e3ece7e225a7b50ed99: Status 404 returned error can't find the container with id be9ccc983877625182a1d7b0d806afdeb3d10049e7429e3ece7e225a7b50ed99 Sep 16 19:23:54.195078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:54.195036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" event={"ID":"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b","Type":"ContainerStarted","Data":"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720"} Sep 16 19:23:54.195533 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:54.195123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" event={"ID":"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b","Type":"ContainerStarted","Data":"be9ccc983877625182a1d7b0d806afdeb3d10049e7429e3ece7e225a7b50ed99"} Sep 16 19:23:54.225970 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:54.225913 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" podStartSLOduration=2.225892872 podStartE2EDuration="2.225892872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:54.223604363 +0000 UTC m=+1118.429560126" watchObservedRunningTime="2026-09-16 19:23:54.225892872 +0000 UTC m=+1118.431848627" Sep 16 19:23:55.156923 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:55.156893 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:57.099508 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.099469 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:23:57.222951 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:57.222912 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:23:57.222951 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:57.222951 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:57.222951 ip-10-0-128-148 kubenswrapper[2565]: > containerID="e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" cmd=["/bin/sleep","15"] Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.222964 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" message="" err=< Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:57.222977 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerName="main" containerID="cri-o://e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" Sep 16 19:23:57.223314 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.223028 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerName="main" containerID="cri-o://e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" gracePeriod=60 Sep 16 19:23:57.480573 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.480550 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:57.557072 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557044 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557072 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557077 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4snjm\" (UniqueName: \"kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557123 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557164 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home\") pod \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\" (UID: \"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b\") " Sep 16 19:23:57.557525 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557380 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:57.557525 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:57.557642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557585 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:57.557642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557605 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:57.557642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.557609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home" (OuterVolumeSpecName: "home") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:57.559622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.559593 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm" (OuterVolumeSpecName: "kube-api-access-4snjm") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "kube-api-access-4snjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:23:57.559622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.559592 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:23:57.559761 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.559635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm" (OuterVolumeSpecName: "dshm") pod "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" (UID: "3f5ac0ed-a593-4be6-a662-7bd2dc35d39b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:23:57.659045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.658964 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:57.659045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.658993 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snjm\" (UniqueName: \"kubernetes.io/projected/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-kube-api-access-4snjm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:57.659045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.659003 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:57.659045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:57.659011 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:23:58.222031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.221992 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerID="e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" exitCode=0 Sep 16 19:23:58.222500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.222037 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" Sep 16 19:23:58.222500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.222064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" event={"ID":"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b","Type":"ContainerDied","Data":"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720"} Sep 16 19:23:58.222500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.222113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf" event={"ID":"3f5ac0ed-a593-4be6-a662-7bd2dc35d39b","Type":"ContainerDied","Data":"be9ccc983877625182a1d7b0d806afdeb3d10049e7429e3ece7e225a7b50ed99"} Sep 16 19:23:58.222500 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.222134 2565 scope.go:117] "RemoveContainer" containerID="e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" Sep 16 19:23:58.233325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.233165 2565 scope.go:117] "RemoveContainer" containerID="e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" Sep 16 19:23:58.233488 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:23:58.233461 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720\": container with ID starting with e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720 not found: ID does not exist" containerID="e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720" Sep 16 19:23:58.233534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.233499 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720"} err="failed to get container status \"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720\": rpc error: code = NotFound desc = could not find container \"e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720\": container with ID starting with e4b008ec7d9104a311fc40de99a7f6b0c962eac7962f6e545f9f73a0f5cf7720 not found: ID does not exist" Sep 16 19:23:58.261215 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.261183 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:23:58.261215 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:23:58.261222 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf869hf"] Sep 16 19:24:00.253622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:00.253588 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" path="/var/lib/kubelet/pods/3f5ac0ed-a593-4be6-a662-7bd2dc35d39b/volumes" Sep 16 19:24:07.512677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.512644 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:24:07.513084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.513050 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerName="main" Sep 16 19:24:07.513084 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.513061 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerName="main" Sep 16 19:24:07.513157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.513132 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5ac0ed-a593-4be6-a662-7bd2dc35d39b" containerName="main" Sep 16 19:24:07.517561 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.517544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.521879 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.521856 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-k7r8s\"" Sep 16 19:24:07.522563 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.522542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\"" Sep 16 19:24:07.522692 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.522672 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-gr6dd\"" Sep 16 19:24:07.522744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.522713 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\"" Sep 16 19:24:07.522744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.522725 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\"" Sep 16 19:24:07.533418 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.533392 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:24:07.654644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.654610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbj5\" (UniqueName: \"kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.654811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.654650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.756130 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.756091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbj5\" (UniqueName: \"kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.756130 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.756135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.758949 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.758916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.771556 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.771503 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbj5\" (UniqueName: \"kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5\") pod \"auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:07.828362 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:07.828329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:08.121775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:08.121743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:24:08.123141 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:24:08.123120 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50a0556_ddfa_4549_a501_539753f4fb5f.slice/crio-c0d4ae8e08bfc26d4ba7f48bf25269a09a02252bd7957408fc40f476bfc30ee1 WatchSource:0}: Error finding container c0d4ae8e08bfc26d4ba7f48bf25269a09a02252bd7957408fc40f476bfc30ee1: Status 404 returned error can't find the container with id c0d4ae8e08bfc26d4ba7f48bf25269a09a02252bd7957408fc40f476bfc30ee1 Sep 16 19:24:08.281478 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:08.281427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" event={"ID":"d50a0556-ddfa-4549-a501-539753f4fb5f","Type":"ContainerStarted","Data":"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508"} Sep 16 19:24:08.281668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:08.281493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" event={"ID":"d50a0556-ddfa-4549-a501-539753f4fb5f","Type":"ContainerStarted","Data":"c0d4ae8e08bfc26d4ba7f48bf25269a09a02252bd7957408fc40f476bfc30ee1"} Sep 16 19:24:08.281668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:08.281537 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:24:10.224370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.224271 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podStartSLOduration=3.224250864 podStartE2EDuration="3.224250864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:08.325958811 +0000 UTC m=+1132.531914565" watchObservedRunningTime="2026-09-16 19:24:10.224250864 +0000 UTC m=+1134.430206617" Sep 16 19:24:10.226605 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.226583 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:24:10.235935 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:24:10.235900 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:24:10.235935 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:24:10.235935 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:24:10.235935 ip-10-0-128-148 kubenswrapper[2565]: > containerID="7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" cmd=["/bin/sleep","15"] Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.235955 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" message="" err=< Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:24:10.235967 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerName="main" containerID="cri-o://7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" Sep 16 19:24:10.236146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.236013 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerName="main" containerID="cri-o://7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" gracePeriod=60 Sep 16 19:24:10.505469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.505444 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:24:10.681390 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681589 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbcxr\" (UniqueName: \"kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681633 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs\") pod \"e614fa46-3d00-4520-8d12-78a968ff32c5\" (UID: \"e614fa46-3d00-4520-8d12-78a968ff32c5\") " Sep 16 19:24:10.681821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681671 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:24:10.681821 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681782 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home" (OuterVolumeSpecName: "home") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:24:10.682109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:24:10.682109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681958 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:10.682109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.681976 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:10.682109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.682001 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:10.684595 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.684563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:24:10.684720 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.684619 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm" (OuterVolumeSpecName: "dshm") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:24:10.684720 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.684658 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr" (OuterVolumeSpecName: "kube-api-access-gbcxr") pod "e614fa46-3d00-4520-8d12-78a968ff32c5" (UID: "e614fa46-3d00-4520-8d12-78a968ff32c5"). InnerVolumeSpecName "kube-api-access-gbcxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:24:10.783408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.783305 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e614fa46-3d00-4520-8d12-78a968ff32c5-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:10.783408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.783338 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcxr\" (UniqueName: \"kubernetes.io/projected/e614fa46-3d00-4520-8d12-78a968ff32c5-kube-api-access-gbcxr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:10.783408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:10.783352 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e614fa46-3d00-4520-8d12-78a968ff32c5-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:24:11.301470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.301438 2565 generic.go:356] "Generic (PLEG): container finished" podID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerID="7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" exitCode=0 Sep 16 19:24:11.301913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.301489 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" Sep 16 19:24:11.301913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.301539 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" event={"ID":"e614fa46-3d00-4520-8d12-78a968ff32c5","Type":"ContainerDied","Data":"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba"} Sep 16 19:24:11.301913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.301588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9" event={"ID":"e614fa46-3d00-4520-8d12-78a968ff32c5","Type":"ContainerDied","Data":"bd5559161ee05a88881cebdf963339f42645aa36093e724983619741cae2e4da"} Sep 16 19:24:11.301913 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.301612 2565 scope.go:117] "RemoveContainer" containerID="7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" Sep 16 19:24:11.312152 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.312133 2565 scope.go:117] "RemoveContainer" containerID="7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" Sep 16 19:24:11.312461 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:24:11.312441 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba\": container with ID starting with 7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba not found: ID does not exist" containerID="7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba" Sep 16 19:24:11.312515 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.312471 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba"} err="failed to get container status \"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba\": rpc error: code = NotFound desc = could not find container \"7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba\": container with ID starting with 7e29ff8a85ce94cdd08b8d15e980bbbe3df3131605ec218e9371248907104dba not found: ID does not exist" Sep 16 19:24:11.331743 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.331722 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:24:11.336945 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:11.336923 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6ftv9"] Sep 16 19:24:12.254553 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:12.254512 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" path="/var/lib/kubelet/pods/e614fa46-3d00-4520-8d12-78a968ff32c5/volumes" Sep 16 19:24:39.289937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:24:39.289848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:25:03.183969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.183931 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:03.184479 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.184463 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerName="main" Sep 16 19:25:03.184529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.184482 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerName="main" Sep 16 19:25:03.184565 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.184545 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e614fa46-3d00-4520-8d12-78a968ff32c5" containerName="main" Sep 16 19:25:03.187505 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.187489 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.192135 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.192113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\"" Sep 16 19:25:03.192269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.192182 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\"" Sep 16 19:25:03.192467 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.192449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-l9kb6\"" Sep 16 19:25:03.192521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.192505 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Sep 16 19:25:03.200061 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.200037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:03.327582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.327767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.327767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.327767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.327905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56ck\" (UniqueName: \"kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.327905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.327835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56ck\" (UniqueName: \"kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.428918 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.428912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.429527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.429364 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.429527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.429400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.429527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.429481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.431629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.431594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.431959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.431938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.453469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.453389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56ck\" (UniqueName: \"kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.498564 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.498523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:03.819954 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:03.819930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:03.822354 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:25:03.822327 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb93be64_4cd0_4e36_981a_4aae072f7152.slice/crio-51327c3894c6f5cdc4e162a047583015d6155d1d56567784419998cb243c4772 WatchSource:0}: Error finding container 51327c3894c6f5cdc4e162a047583015d6155d1d56567784419998cb243c4772: Status 404 returned error can't find the container with id 51327c3894c6f5cdc4e162a047583015d6155d1d56567784419998cb243c4772 Sep 16 19:25:04.608082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:04.608042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" event={"ID":"cb93be64-4cd0-4e36-981a-4aae072f7152","Type":"ContainerStarted","Data":"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9"} Sep 16 19:25:04.608621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:04.608092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" event={"ID":"cb93be64-4cd0-4e36-981a-4aae072f7152","Type":"ContainerStarted","Data":"51327c3894c6f5cdc4e162a047583015d6155d1d56567784419998cb243c4772"} Sep 16 19:25:04.635897 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:04.635831 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" podStartSLOduration=1.635810693 podStartE2EDuration="1.635810693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:04.634953736 +0000 UTC m=+1188.840909490" watchObservedRunningTime="2026-09-16 19:25:04.635810693 +0000 UTC m=+1188.841766449" Sep 16 19:25:05.499553 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:05.499515 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:13.498907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:13.498870 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:13.511573 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:13.511548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:13.672820 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:13.672785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:37.720868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:37.720835 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:37.730233 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:25:37.730193 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:25:37.730233 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:25:37.730233 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:25:37.730233 ip-10-0-128-148 kubenswrapper[2565]: > containerID="97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" cmd=["/bin/sleep","15"] Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:37.730248 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" message="" err=< Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:25:37.730261 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerName="main" containerID="cri-o://97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" Sep 16 19:25:37.730564 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:37.730323 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerName="main" containerID="cri-o://97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" gracePeriod=60 Sep 16 19:25:38.071964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.071941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:38.138898 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.138862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.138935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.138968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.138993 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139021 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139096 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139054 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56ck\" (UniqueName: \"kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck\") pod \"cb93be64-4cd0-4e36-981a-4aae072f7152\" (UID: \"cb93be64-4cd0-4e36-981a-4aae072f7152\") " Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139156 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home" (OuterVolumeSpecName: "home") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139204 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139349 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139361 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.139374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.139371 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.141380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.141345 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:25:38.141496 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.141380 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm" (OuterVolumeSpecName: "dshm") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:25:38.141496 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.141416 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck" (OuterVolumeSpecName: "kube-api-access-p56ck") pod "cb93be64-4cd0-4e36-981a-4aae072f7152" (UID: "cb93be64-4cd0-4e36-981a-4aae072f7152"). InnerVolumeSpecName "kube-api-access-p56ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:25:38.239968 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.239931 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb93be64-4cd0-4e36-981a-4aae072f7152-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.239968 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.239964 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56ck\" (UniqueName: \"kubernetes.io/projected/cb93be64-4cd0-4e36-981a-4aae072f7152-kube-api-access-p56ck\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.240181 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.239979 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb93be64-4cd0-4e36-981a-4aae072f7152-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:25:38.807787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.807752 2565 generic.go:356] "Generic (PLEG): container finished" podID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerID="97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" exitCode=0 Sep 16 19:25:38.808251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.807798 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" Sep 16 19:25:38.808251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.807850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" event={"ID":"cb93be64-4cd0-4e36-981a-4aae072f7152","Type":"ContainerDied","Data":"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9"} Sep 16 19:25:38.808251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.807904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz" event={"ID":"cb93be64-4cd0-4e36-981a-4aae072f7152","Type":"ContainerDied","Data":"51327c3894c6f5cdc4e162a047583015d6155d1d56567784419998cb243c4772"} Sep 16 19:25:38.808251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.807927 2565 scope.go:117] "RemoveContainer" containerID="97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" Sep 16 19:25:38.817585 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.817539 2565 scope.go:117] "RemoveContainer" containerID="97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" Sep 16 19:25:38.817859 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:25:38.817838 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9\": container with ID starting with 97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9 not found: ID does not exist" containerID="97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9" Sep 16 19:25:38.817859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.817867 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9"} err="failed to get container status \"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9\": rpc error: code = NotFound desc = could not find container \"97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9\": container with ID starting with 97f284c86a337d5ce4c1a12edc859ac4c8fa376ed6737bd5b7e5ae49eb8e0bc9 not found: ID does not exist" Sep 16 19:25:38.832747 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.832716 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:38.838195 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:38.838166 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94frphzz"] Sep 16 19:25:40.254886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:25:40.254852 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" path="/var/lib/kubelet/pods/cb93be64-4cd0-4e36-981a-4aae072f7152/volumes" Sep 16 19:26:12.191654 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:12.191367 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:26:19.289481 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:19.289437 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:26:27.213350 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:27.213301 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" containerID="cri-o://573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508" gracePeriod=45 Sep 16 19:26:29.289461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:29.289425 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:26:33.698815 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.698778 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:26:33.699185 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.699166 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerName="main" Sep 16 19:26:33.699185 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.699177 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerName="main" Sep 16 19:26:33.699269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.699260 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb93be64-4cd0-4e36-981a-4aae072f7152" containerName="main" Sep 16 19:26:33.703731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.703713 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.714751 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.714731 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\"" Sep 16 19:26:33.714872 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.714754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\"" Sep 16 19:26:33.714872 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.714815 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tgtvg\"" Sep 16 19:26:33.714872 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.714754 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\"" Sep 16 19:26:33.731145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.731115 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:26:33.822982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.822941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.822982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.822981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.823211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.823067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.823211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.823106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj94j\" (UniqueName: \"kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.823211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.823125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.823211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.823171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924192 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924152 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj94j\" (UniqueName: \"kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.924771 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.924720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.926738 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.926719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.926982 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.926964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:33.936631 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:33.936601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj94j\" (UniqueName: \"kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:34.014214 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:34.014182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:34.304038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:34.304012 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:26:34.305653 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:26:34.305621 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1cccd6_645f_4ffa_8f87_72460d37eec3.slice/crio-9929d6f051ab35bb8bea245685a8c3398b187a82bf7338c675b758974b42b793 WatchSource:0}: Error finding container 9929d6f051ab35bb8bea245685a8c3398b187a82bf7338c675b758974b42b793: Status 404 returned error can't find the container with id 9929d6f051ab35bb8bea245685a8c3398b187a82bf7338c675b758974b42b793 Sep 16 19:26:35.130829 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:35.130785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" event={"ID":"1f1cccd6-645f-4ffa-8f87-72460d37eec3","Type":"ContainerStarted","Data":"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c"} Sep 16 19:26:35.131237 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:35.130847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" event={"ID":"1f1cccd6-645f-4ffa-8f87-72460d37eec3","Type":"ContainerStarted","Data":"9929d6f051ab35bb8bea245685a8c3398b187a82bf7338c675b758974b42b793"} Sep 16 19:26:35.187962 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:35.187896 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" podStartSLOduration=2.18787737 podStartE2EDuration="2.18787737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:35.187094691 +0000 UTC m=+1279.393050443" watchObservedRunningTime="2026-09-16 19:26:35.18787737 +0000 UTC m=+1279.393833146" Sep 16 19:26:36.014715 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:36.014678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:39.289669 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:39.289631 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:26:39.290136 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:39.289760 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:26:44.015227 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:44.015189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:44.027807 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:44.027776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:44.211966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:44.211938 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:26:49.289882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:49.289836 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:26:57.475011 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.474950 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:26:57.639105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.639066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs\") pod \"d50a0556-ddfa-4549-a501-539753f4fb5f\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " Sep 16 19:26:57.639332 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.639185 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvbj5\" (UniqueName: \"kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5\") pod \"d50a0556-ddfa-4549-a501-539753f4fb5f\" (UID: \"d50a0556-ddfa-4549-a501-539753f4fb5f\") " Sep 16 19:26:57.641587 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.641560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d50a0556-ddfa-4549-a501-539753f4fb5f" (UID: "d50a0556-ddfa-4549-a501-539753f4fb5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:26:57.641587 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.641569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5" (OuterVolumeSpecName: "kube-api-access-wvbj5") pod "d50a0556-ddfa-4549-a501-539753f4fb5f" (UID: "d50a0556-ddfa-4549-a501-539753f4fb5f"). InnerVolumeSpecName "kube-api-access-wvbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:26:57.739956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.739867 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbj5\" (UniqueName: \"kubernetes.io/projected/d50a0556-ddfa-4549-a501-539753f4fb5f-kube-api-access-wvbj5\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:26:57.739956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:57.739898 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50a0556-ddfa-4549-a501-539753f4fb5f-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:26:58.281812 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.281772 2565 generic.go:356] "Generic (PLEG): container finished" podID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerID="573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508" exitCode=0 Sep 16 19:26:58.282001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.281819 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" Sep 16 19:26:58.282001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.281859 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" event={"ID":"d50a0556-ddfa-4549-a501-539753f4fb5f","Type":"ContainerDied","Data":"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508"} Sep 16 19:26:58.282001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.281915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q" event={"ID":"d50a0556-ddfa-4549-a501-539753f4fb5f","Type":"ContainerDied","Data":"c0d4ae8e08bfc26d4ba7f48bf25269a09a02252bd7957408fc40f476bfc30ee1"} Sep 16 19:26:58.282001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.281937 2565 scope.go:117] "RemoveContainer" containerID="573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508" Sep 16 19:26:58.292110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.292089 2565 scope.go:117] "RemoveContainer" containerID="573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508" Sep 16 19:26:58.292416 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:26:58.292396 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508\": container with ID starting with 573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508 not found: ID does not exist" containerID="573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508" Sep 16 19:26:58.292478 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.292425 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508"} err="failed to get container status \"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508\": rpc error: code = NotFound desc = could not find container \"573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508\": container with ID starting with 573d5cd102184f91c610eb30d69b442a461347c23df7ca92f480942c1e462508 not found: ID does not exist" Sep 16 19:26:58.304485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.304460 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:26:58.310387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:26:58.310366 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-7b679bf4d6-6xb2q"] Sep 16 19:27:00.259600 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:00.259568 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" path="/var/lib/kubelet/pods/d50a0556-ddfa-4549-a501-539753f4fb5f/volumes" Sep 16 19:27:20.830514 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.830481 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:27:20.831019 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.830890 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" Sep 16 19:27:20.831019 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.830901 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" Sep 16 19:27:20.831019 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.830974 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50a0556-ddfa-4549-a501-539753f4fb5f" containerName="main" Sep 16 19:27:20.835384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.835361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:20.839001 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.838973 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\"" Sep 16 19:27:20.839142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.838973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-bzhlb\"" Sep 16 19:27:20.839142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.839031 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\"" Sep 16 19:27:20.839350 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.839336 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-jk2qd\"" Sep 16 19:27:20.839437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.839421 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\"" Sep 16 19:27:20.845735 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.845712 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:27:20.933855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.933821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcj4\" (UniqueName: \"kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:20.933855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:20.933865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.034678 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.034639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcj4\" (UniqueName: \"kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.034885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.034694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.037511 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.037477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.049683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.049654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcj4\" (UniqueName: \"kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4\") pod \"auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.147200 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.147118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:21.433664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:21.433639 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:27:21.435549 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:27:21.435525 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4178e289_fa39_40fc_83d7_a377d24ca2af.slice/crio-97c97a18f37c0b6020ce29992aa05c1d23f971697b82d9cdc907710fc1c4fa28 WatchSource:0}: Error finding container 97c97a18f37c0b6020ce29992aa05c1d23f971697b82d9cdc907710fc1c4fa28: Status 404 returned error can't find the container with id 97c97a18f37c0b6020ce29992aa05c1d23f971697b82d9cdc907710fc1c4fa28 Sep 16 19:27:22.423201 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:22.423163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" event={"ID":"4178e289-fa39-40fc-83d7-a377d24ca2af","Type":"ContainerStarted","Data":"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea"} Sep 16 19:27:22.423668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:22.423229 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" event={"ID":"4178e289-fa39-40fc-83d7-a377d24ca2af","Type":"ContainerStarted","Data":"97c97a18f37c0b6020ce29992aa05c1d23f971697b82d9cdc907710fc1c4fa28"} Sep 16 19:27:22.423668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:22.423250 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:27:23.347686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.347622 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podStartSLOduration=3.347605736 podStartE2EDuration="3.347605736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:22.458651442 +0000 UTC m=+1326.664607195" watchObservedRunningTime="2026-09-16 19:27:23.347605736 +0000 UTC m=+1327.553561488" Sep 16 19:27:23.349819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.349796 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:27:23.359619 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:27:23.359579 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:27:23.359619 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:27:23.359619 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:27:23.359619 ip-10-0-128-148 kubenswrapper[2565]: > containerID="fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" cmd=["/bin/sleep","15"] Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.359639 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" message="" err=< Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:27:23.359651 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerName="main" containerID="cri-o://fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" Sep 16 19:27:23.359851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.359695 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerName="main" containerID="cri-o://fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" gracePeriod=60 Sep 16 19:27:23.715866 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.715841 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:27:23.759157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759177 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759204 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj94j\" (UniqueName: \"kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759254 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759317 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home\") pod \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\" (UID: \"1f1cccd6-645f-4ffa-8f87-72460d37eec3\") " Sep 16 19:27:23.759619 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759559 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:27:23.759619 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759572 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:27:23.759698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.759635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home" (OuterVolumeSpecName: "home") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:27:23.761696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.761675 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm" (OuterVolumeSpecName: "dshm") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:27:23.761805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.761725 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j" (OuterVolumeSpecName: "kube-api-access-gj94j") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "kube-api-access-gj94j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:27:23.761805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.761742 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f1cccd6-645f-4ffa-8f87-72460d37eec3" (UID: "1f1cccd6-645f-4ffa-8f87-72460d37eec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:27:23.860884 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860854 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:23.860884 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860882 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:23.860884 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860893 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj94j\" (UniqueName: \"kubernetes.io/projected/1f1cccd6-645f-4ffa-8f87-72460d37eec3-kube-api-access-gj94j\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:23.861120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860902 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:23.861120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860910 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:23.861120 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:23.860918 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f1cccd6-645f-4ffa-8f87-72460d37eec3-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:27:24.435504 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.435463 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerID="fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" exitCode=0 Sep 16 19:27:24.435674 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.435512 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" Sep 16 19:27:24.435674 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.435512 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" event={"ID":"1f1cccd6-645f-4ffa-8f87-72460d37eec3","Type":"ContainerDied","Data":"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c"} Sep 16 19:27:24.435674 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.435568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7" event={"ID":"1f1cccd6-645f-4ffa-8f87-72460d37eec3","Type":"ContainerDied","Data":"9929d6f051ab35bb8bea245685a8c3398b187a82bf7338c675b758974b42b793"} Sep 16 19:27:24.435674 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.435596 2565 scope.go:117] "RemoveContainer" containerID="fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" Sep 16 19:27:24.445135 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.445115 2565 scope.go:117] "RemoveContainer" containerID="fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" Sep 16 19:27:24.445489 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:27:24.445468 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c\": container with ID starting with fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c not found: ID does not exist" containerID="fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c" Sep 16 19:27:24.445547 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.445498 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c"} err="failed to get container status \"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c\": rpc error: code = NotFound desc = could not find container \"fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c\": container with ID starting with fae9c1770b1aef7fc5d8e23399f1dbeaf02d827619f78b388eb1e8040d69042c not found: ID does not exist" Sep 16 19:27:24.463013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.462980 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:27:24.466550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:24.466529 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vtqp7"] Sep 16 19:27:26.254105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:26.254073 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" path="/var/lib/kubelet/pods/1f1cccd6-645f-4ffa-8f87-72460d37eec3/volumes" Sep 16 19:27:53.430384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:27:53.430355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:28:23.497534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.497500 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:28:23.498137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.498048 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerName="main" Sep 16 19:28:23.498137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.498067 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerName="main" Sep 16 19:28:23.498272 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.498176 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1cccd6-645f-4ffa-8f87-72460d37eec3" containerName="main" Sep 16 19:28:23.502422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.502406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.505481 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.505459 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\"" Sep 16 19:28:23.506210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.506190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-llfn9\"" Sep 16 19:28:23.506277 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.506260 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\"" Sep 16 19:28:23.506342 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.506326 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Sep 16 19:28:23.517986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.517946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:28:23.602170 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.602410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.602410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.602410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.602410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602313 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skk7\" (UniqueName: \"kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.602410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.602377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703306 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703408 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skk7\" (UniqueName: \"kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703771 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.703813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.703774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.705858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.705828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.706049 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.706031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.725572 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.725546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skk7\" (UniqueName: \"kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-wg48r\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.812232 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.812148 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:28:23.812416 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.812239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:23.815742 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.815559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:23.818979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.818958 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-chsqs\"" Sep 16 19:28:23.839160 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.839129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:28:23.905068 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.905037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:23.905224 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:23.905083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bdm\" (UniqueName: \"kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.006111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.006070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bdm\" (UniqueName: \"kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.006351 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.006197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.008906 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.008880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.018018 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.017994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bdm\" (UniqueName: \"kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.099545 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.099513 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:28:24.101065 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:28:24.101030 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c43f6b_0f28_4bde_99e1_49abcb425600.slice/crio-1a35f138f3627f2fee6ea1246949120eb77fdbba4d0c0ea150cad1a88de059c7 WatchSource:0}: Error finding container 1a35f138f3627f2fee6ea1246949120eb77fdbba4d0c0ea150cad1a88de059c7: Status 404 returned error can't find the container with id 1a35f138f3627f2fee6ea1246949120eb77fdbba4d0c0ea150cad1a88de059c7 Sep 16 19:28:24.146102 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.146075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.448401 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.448372 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:28:24.449966 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:28:24.449924 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56744d29_c029_4098_a244_29be77a0f106.slice/crio-453ba60ea3f3bf1fd54379a64f0b48dddbf8d7e3a3c66ac7a6c6c1846d0f99b7 WatchSource:0}: Error finding container 453ba60ea3f3bf1fd54379a64f0b48dddbf8d7e3a3c66ac7a6c6c1846d0f99b7: Status 404 returned error can't find the container with id 453ba60ea3f3bf1fd54379a64f0b48dddbf8d7e3a3c66ac7a6c6c1846d0f99b7 Sep 16 19:28:24.771444 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.771409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" event={"ID":"56744d29-c029-4098-a244-29be77a0f106","Type":"ContainerStarted","Data":"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e"} Sep 16 19:28:24.771946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.771469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" event={"ID":"56744d29-c029-4098-a244-29be77a0f106","Type":"ContainerStarted","Data":"453ba60ea3f3bf1fd54379a64f0b48dddbf8d7e3a3c66ac7a6c6c1846d0f99b7"} Sep 16 19:28:24.771946 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.771601 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:28:24.772998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.772973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" event={"ID":"90c43f6b-0f28-4bde-99e1-49abcb425600","Type":"ContainerStarted","Data":"205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167"} Sep 16 19:28:24.773112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.773021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" event={"ID":"90c43f6b-0f28-4bde-99e1-49abcb425600","Type":"ContainerStarted","Data":"1a35f138f3627f2fee6ea1246949120eb77fdbba4d0c0ea150cad1a88de059c7"} Sep 16 19:28:24.791321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.791231 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podStartSLOduration=1.791210916 podStartE2EDuration="1.791210916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:24.789104082 +0000 UTC m=+1388.995059839" watchObservedRunningTime="2026-09-16 19:28:24.791210916 +0000 UTC m=+1388.997166670" Sep 16 19:28:24.812736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.812703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:24.816050 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:24.816003 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" podStartSLOduration=1.815989492 podStartE2EDuration="1.815989492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:24.813869803 +0000 UTC m=+1389.019825558" watchObservedRunningTime="2026-09-16 19:28:24.815989492 +0000 UTC m=+1389.021945244" Sep 16 19:28:33.813274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:33.813227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:33.826366 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:33.826340 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:33.837723 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:33.837697 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:55.781470 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:55.781418 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:28:57.706089 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.706051 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:28:57.707669 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.707614 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:28:57.716634 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.716606 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:28:57.727087 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:28:57.727045 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:28:57.727087 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:28:57.727087 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:28:57.727087 ip-10-0-128-148 kubenswrapper[2565]: > containerID="205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167" cmd=["/bin/sleep","15"] Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.727108 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" message="" err=< Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:28:57.727127 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" containerID="cri-o://205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167" Sep 16 19:28:57.727433 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.727173 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" containerID="cri-o://205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167" gracePeriod=60 Sep 16 19:28:57.827068 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.827029 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused" Sep 16 19:28:57.965960 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.965927 2565 generic.go:356] "Generic (PLEG): container finished" podID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerID="205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167" exitCode=0 Sep 16 19:28:57.966108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:57.965997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" event={"ID":"90c43f6b-0f28-4bde-99e1-49abcb425600","Type":"ContainerDied","Data":"205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167"} Sep 16 19:28:58.090211 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.090188 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:58.114521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2skk7\" (UniqueName: \"kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114729 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114546 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114729 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114670 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:28:58.114831 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114940 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114856 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home\") pod \"90c43f6b-0f28-4bde-99e1-49abcb425600\" (UID: \"90c43f6b-0f28-4bde-99e1-49abcb425600\") " Sep 16 19:28:58.114940 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.114908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache" (OuterVolumeSpecName: "model-cache") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:28:58.115126 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.115102 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home" (OuterVolumeSpecName: "home") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:28:58.115205 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.115110 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.115324 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.115261 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.116878 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.116859 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm" (OuterVolumeSpecName: "dshm") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:28:58.117038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.116893 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7" (OuterVolumeSpecName: "kube-api-access-2skk7") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "kube-api-access-2skk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:28:58.117038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.116901 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90c43f6b-0f28-4bde-99e1-49abcb425600" (UID: "90c43f6b-0f28-4bde-99e1-49abcb425600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:28:58.216509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.216424 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.216509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.216455 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skk7\" (UniqueName: \"kubernetes.io/projected/90c43f6b-0f28-4bde-99e1-49abcb425600-kube-api-access-2skk7\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.216509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.216466 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c43f6b-0f28-4bde-99e1-49abcb425600-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.216509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.216474 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c43f6b-0f28-4bde-99e1-49abcb425600-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:28:58.973124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.973092 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" Sep 16 19:28:58.973124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.973107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r" event={"ID":"90c43f6b-0f28-4bde-99e1-49abcb425600","Type":"ContainerDied","Data":"1a35f138f3627f2fee6ea1246949120eb77fdbba4d0c0ea150cad1a88de059c7"} Sep 16 19:28:58.973650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:58.973170 2565 scope.go:117] "RemoveContainer" containerID="205a67d7827b166389ade68d643c30f1da6084ad3ae44dacc7ebb2b2a7ed0167" Sep 16 19:28:59.011150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:59.011119 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:28:59.018185 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:28:59.018162 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-wg48r"] Sep 16 19:29:00.254224 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:00.254191 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" path="/var/lib/kubelet/pods/90c43f6b-0f28-4bde-99e1-49abcb425600/volumes" Sep 16 19:29:07.707949 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:07.707856 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:12.411487 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:12.411453 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:29:12.728762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:12.728720 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" containerID="cri-o://94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e" gracePeriod=45 Sep 16 19:29:13.431066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:13.431021 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:17.707747 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:17.707709 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:23.430206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:23.430163 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:27.431434 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:27.431375 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" containerID="cri-o://b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea" gracePeriod=45 Sep 16 19:29:27.708187 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:27.708094 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:33.430451 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:33.430411 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:33.430897 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:33.430543 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:29:37.708653 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:37.708616 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:42.980854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:42.980830 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:29:43.120196 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.120106 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8bdm\" (UniqueName: \"kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm\") pod \"56744d29-c029-4098-a244-29be77a0f106\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " Sep 16 19:29:43.120196 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.120184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs\") pod \"56744d29-c029-4098-a244-29be77a0f106\" (UID: \"56744d29-c029-4098-a244-29be77a0f106\") " Sep 16 19:29:43.122444 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.122418 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56744d29-c029-4098-a244-29be77a0f106" (UID: "56744d29-c029-4098-a244-29be77a0f106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:29:43.122558 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.122475 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm" (OuterVolumeSpecName: "kube-api-access-n8bdm") pod "56744d29-c029-4098-a244-29be77a0f106" (UID: "56744d29-c029-4098-a244-29be77a0f106"). InnerVolumeSpecName "kube-api-access-n8bdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:29:43.221137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.221106 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56744d29-c029-4098-a244-29be77a0f106-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:29:43.221137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.221131 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8bdm\" (UniqueName: \"kubernetes.io/projected/56744d29-c029-4098-a244-29be77a0f106-kube-api-access-n8bdm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:29:43.225799 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.225772 2565 generic.go:356] "Generic (PLEG): container finished" podID="56744d29-c029-4098-a244-29be77a0f106" containerID="94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e" exitCode=0 Sep 16 19:29:43.225969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.225802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" event={"ID":"56744d29-c029-4098-a244-29be77a0f106","Type":"ContainerDied","Data":"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e"} Sep 16 19:29:43.225969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.225831 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" Sep 16 19:29:43.225969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.225860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5" event={"ID":"56744d29-c029-4098-a244-29be77a0f106","Type":"ContainerDied","Data":"453ba60ea3f3bf1fd54379a64f0b48dddbf8d7e3a3c66ac7a6c6c1846d0f99b7"} Sep 16 19:29:43.225969 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.225882 2565 scope.go:117] "RemoveContainer" containerID="94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e" Sep 16 19:29:43.235787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.235769 2565 scope.go:117] "RemoveContainer" containerID="94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e" Sep 16 19:29:43.236052 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:29:43.236033 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e\": container with ID starting with 94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e not found: ID does not exist" containerID="94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e" Sep 16 19:29:43.236101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.236062 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e"} err="failed to get container status \"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e\": rpc error: code = NotFound desc = could not find container \"94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e\": container with ID starting with 94adc23a9768e42da429dab47b611409d2145d92a3a30396d2e4f863654c9c1e not found: ID does not exist" Sep 16 19:29:43.252397 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.252367 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:29:43.256480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.256458 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5d68b67q5dn5"] Sep 16 19:29:43.430964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:43.430882 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:44.253819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:44.253788 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56744d29-c029-4098-a244-29be77a0f106" path="/var/lib/kubelet/pods/56744d29-c029-4098-a244-29be77a0f106/volumes" Sep 16 19:29:50.225380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225342 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:29:50.225766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225753 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" Sep 16 19:29:50.225766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225765 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" Sep 16 19:29:50.225844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225777 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" Sep 16 19:29:50.225844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225783 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" Sep 16 19:29:50.225901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225851 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c43f6b-0f28-4bde-99e1-49abcb425600" containerName="main" Sep 16 19:29:50.225901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.225860 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="56744d29-c029-4098-a244-29be77a0f106" containerName="main" Sep 16 19:29:50.229031 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.229009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.245588 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.245566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-5cwx5\"" Sep 16 19:29:50.245702 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.245589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\"" Sep 16 19:29:50.257566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.257545 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\"" Sep 16 19:29:50.259802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.259786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\"" Sep 16 19:29:50.276485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcdk6\" (UniqueName: \"kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.276604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.276604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.276604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.276604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.276743 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.276626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.298115 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.298085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:29:50.377791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.377755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.377974 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.377801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcdk6\" (UniqueName: \"kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.377974 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.377923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.377986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.378021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.378046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.378359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.378397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.378491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.378429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.380188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.380160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.380461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.380445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.433576 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.433548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcdk6\" (UniqueName: \"kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nz8wh\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.539013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.538927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:50.840209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:50.840115 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:29:50.848541 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:29:50.848516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c42621e_6a57_482d_8e5b_9ce8fa84784a.slice/crio-3da70a30741c30d1591e9ff430530545f14e7cd35cabcb7e0979646ac14a1da9 WatchSource:0}: Error finding container 3da70a30741c30d1591e9ff430530545f14e7cd35cabcb7e0979646ac14a1da9: Status 404 returned error can't find the container with id 3da70a30741c30d1591e9ff430530545f14e7cd35cabcb7e0979646ac14a1da9 Sep 16 19:29:51.271987 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:51.271954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" event={"ID":"7c42621e-6a57-482d-8e5b-9ce8fa84784a","Type":"ContainerStarted","Data":"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255"} Sep 16 19:29:51.271987 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:51.272004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" event={"ID":"7c42621e-6a57-482d-8e5b-9ce8fa84784a","Type":"ContainerStarted","Data":"3da70a30741c30d1591e9ff430530545f14e7cd35cabcb7e0979646ac14a1da9"} Sep 16 19:29:51.368072 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:51.368014 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" podStartSLOduration=1.367996299 podStartE2EDuration="1.367996299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:51.366361441 +0000 UTC m=+1475.572317208" watchObservedRunningTime="2026-09-16 19:29:51.367996299 +0000 UTC m=+1475.573952051" Sep 16 19:29:51.539466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:51.539380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:29:53.430185 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:53.430147 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:29:57.695762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.695697 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:29:57.750989 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.750955 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs\") pod \"4178e289-fa39-40fc-83d7-a377d24ca2af\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " Sep 16 19:29:57.751177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.751047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgcj4\" (UniqueName: \"kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4\") pod \"4178e289-fa39-40fc-83d7-a377d24ca2af\" (UID: \"4178e289-fa39-40fc-83d7-a377d24ca2af\") " Sep 16 19:29:57.753268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.753244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4" (OuterVolumeSpecName: "kube-api-access-fgcj4") pod "4178e289-fa39-40fc-83d7-a377d24ca2af" (UID: "4178e289-fa39-40fc-83d7-a377d24ca2af"). InnerVolumeSpecName "kube-api-access-fgcj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:29:57.753377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.753263 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4178e289-fa39-40fc-83d7-a377d24ca2af" (UID: "4178e289-fa39-40fc-83d7-a377d24ca2af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:29:57.852668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.852633 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178e289-fa39-40fc-83d7-a377d24ca2af-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:29:57.852668 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:57.852663 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcj4\" (UniqueName: \"kubernetes.io/projected/4178e289-fa39-40fc-83d7-a377d24ca2af-kube-api-access-fgcj4\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:29:58.310494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.310453 2565 generic.go:356] "Generic (PLEG): container finished" podID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerID="b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea" exitCode=0 Sep 16 19:29:58.310666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.310505 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" Sep 16 19:29:58.310666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.310546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" event={"ID":"4178e289-fa39-40fc-83d7-a377d24ca2af","Type":"ContainerDied","Data":"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea"} Sep 16 19:29:58.310666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.310600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz" event={"ID":"4178e289-fa39-40fc-83d7-a377d24ca2af","Type":"ContainerDied","Data":"97c97a18f37c0b6020ce29992aa05c1d23f971697b82d9cdc907710fc1c4fa28"} Sep 16 19:29:58.310666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.310624 2565 scope.go:117] "RemoveContainer" containerID="b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea" Sep 16 19:29:58.330351 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.330330 2565 scope.go:117] "RemoveContainer" containerID="b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea" Sep 16 19:29:58.330677 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:29:58.330655 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea\": container with ID starting with b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea not found: ID does not exist" containerID="b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea" Sep 16 19:29:58.330737 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.330698 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea"} err="failed to get container status \"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea\": rpc error: code = NotFound desc = could not find container \"b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea\": container with ID starting with b5fc44246b9392ae7005feb117cdc75daed8fce0d332883682a0ec303ea6b5ea not found: ID does not exist" Sep 16 19:29:58.337513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.337490 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:29:58.344069 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:29:58.344047 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-67fb9fc4658hbkz"] Sep 16 19:30:00.253911 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:00.253866 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" path="/var/lib/kubelet/pods/4178e289-fa39-40fc-83d7-a377d24ca2af/volumes" Sep 16 19:30:00.539317 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:00.539193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:30:00.551931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:00.551903 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:30:01.341098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:01.341062 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:30:25.185521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.185471 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:30:25.186161 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.186141 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" Sep 16 19:30:25.186209 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.186167 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" Sep 16 19:30:25.186268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.186257 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4178e289-fa39-40fc-83d7-a377d24ca2af" containerName="main" Sep 16 19:30:25.190662 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.190644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.193335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.193312 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\"" Sep 16 19:30:25.194308 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.194275 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-7c8fv\"" Sep 16 19:30:25.194413 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.194393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-zghdq\"" Sep 16 19:30:25.194493 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.194478 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\"" Sep 16 19:30:25.194779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.194761 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\"" Sep 16 19:30:25.208915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.208889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:30:25.291434 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.291398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsth\" (UniqueName: \"kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.291609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.291457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.391931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.391898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsth\" (UniqueName: \"kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.392121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.391956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.394695 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.394674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.402034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.402011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsth\" (UniqueName: \"kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth\") pod \"auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.501264 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.501231 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:25.787792 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:25.787762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:30:25.789694 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:30:25.789670 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c1360_4185_4f41_b9a7_c2b09a729691.slice/crio-317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e WatchSource:0}: Error finding container 317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e: Status 404 returned error can't find the container with id 317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e Sep 16 19:30:26.473915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:26.473880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" event={"ID":"234c1360-4185-4f41-b9a7-c2b09a729691","Type":"ContainerStarted","Data":"f2e67032d53de56a9e9a415df62e160865c75a005a94b6f8374bcf6d28b0edfb"} Sep 16 19:30:26.474330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:26.473936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" event={"ID":"234c1360-4185-4f41-b9a7-c2b09a729691","Type":"ContainerStarted","Data":"317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e"} Sep 16 19:30:26.474330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:26.473978 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:30:26.498744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:26.498687 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podStartSLOduration=1.498664933 podStartE2EDuration="1.498664933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:26.49597777 +0000 UTC m=+1510.701933558" watchObservedRunningTime="2026-09-16 19:30:26.498664933 +0000 UTC m=+1510.704620686" Sep 16 19:30:38.042631 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.042539 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:30:38.052211 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:30:38.052161 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:30:38.052211 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:30:38.052211 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:30:38.052211 ip-10-0-128-148 kubenswrapper[2565]: > containerID="7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" cmd=["/bin/sleep","15"] Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.052228 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" message="" err=< Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:30:38.052243 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" containerID="cri-o://7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" Sep 16 19:30:38.052586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.052330 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" containerID="cri-o://7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" gracePeriod=60 Sep 16 19:30:38.413603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.413571 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:30:38.507882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.507847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.507882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.507889 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.508090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.507927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.508090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.507962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcdk6\" (UniqueName: \"kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.508090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508002 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.508090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home\") pod \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\" (UID: \"7c42621e-6a57-482d-8e5b-9ce8fa84784a\") " Sep 16 19:30:38.508336 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508132 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:30:38.508336 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508228 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:30:38.508446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home" (OuterVolumeSpecName: "home") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:30:38.508446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508425 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:38.508446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.508435 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:38.510510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.510475 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm" (OuterVolumeSpecName: "dshm") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:30:38.510510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.510479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:30:38.510642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.510506 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6" (OuterVolumeSpecName: "kube-api-access-dcdk6") pod "7c42621e-6a57-482d-8e5b-9ce8fa84784a" (UID: "7c42621e-6a57-482d-8e5b-9ce8fa84784a"). InnerVolumeSpecName "kube-api-access-dcdk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:30:38.551696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.551657 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerID="7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" exitCode=0 Sep 16 19:30:38.551885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.551703 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" Sep 16 19:30:38.551885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.551749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" event={"ID":"7c42621e-6a57-482d-8e5b-9ce8fa84784a","Type":"ContainerDied","Data":"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255"} Sep 16 19:30:38.551885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.551813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" event={"ID":"7c42621e-6a57-482d-8e5b-9ce8fa84784a","Type":"ContainerDied","Data":"3da70a30741c30d1591e9ff430530545f14e7cd35cabcb7e0979646ac14a1da9"} Sep 16 19:30:38.551885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.551843 2565 scope.go:117] "RemoveContainer" containerID="7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" Sep 16 19:30:38.562208 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.562190 2565 scope.go:117] "RemoveContainer" containerID="7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" Sep 16 19:30:38.562538 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:30:38.562522 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255\": container with ID starting with 7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255 not found: ID does not exist" containerID="7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255" Sep 16 19:30:38.562609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.562548 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255"} err="failed to get container status \"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255\": rpc error: code = NotFound desc = could not find container \"7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255\": container with ID starting with 7b88a3c305c19b810fc09b3a76f712639347c22de103b0805c3054de5e532255 not found: ID does not exist" Sep 16 19:30:38.576528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.576465 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:30:38.584027 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.584000 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh"] Sep 16 19:30:38.609521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.609486 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcdk6\" (UniqueName: \"kubernetes.io/projected/7c42621e-6a57-482d-8e5b-9ce8fa84784a-kube-api-access-dcdk6\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:38.609521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.609513 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c42621e-6a57-482d-8e5b-9ce8fa84784a-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:38.609521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.609523 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:38.609521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:38.609530 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c42621e-6a57-482d-8e5b-9ce8fa84784a-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:30:39.330744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:39.330696 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nz8wh" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 16 19:30:40.254058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:40.254026 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" path="/var/lib/kubelet/pods/7c42621e-6a57-482d-8e5b-9ce8fa84784a/volumes" Sep 16 19:30:57.481234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:30:57.481205 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:32:30.163762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:30.163654 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:32:37.481105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:37.481067 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:32:45.186169 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:45.186091 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" containerID="cri-o://f2e67032d53de56a9e9a415df62e160865c75a005a94b6f8374bcf6d28b0edfb" gracePeriod=45 Sep 16 19:32:47.481721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:47.481678 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:32:57.481715 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:57.481676 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:32:57.482156 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:32:57.481806 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:33:07.481834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:07.481790 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:33:15.435266 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.435235 2565 generic.go:356] "Generic (PLEG): container finished" podID="234c1360-4185-4f41-b9a7-c2b09a729691" containerID="f2e67032d53de56a9e9a415df62e160865c75a005a94b6f8374bcf6d28b0edfb" exitCode=0 Sep 16 19:33:15.435623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.435332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" event={"ID":"234c1360-4185-4f41-b9a7-c2b09a729691","Type":"ContainerDied","Data":"f2e67032d53de56a9e9a415df62e160865c75a005a94b6f8374bcf6d28b0edfb"} Sep 16 19:33:15.435623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.435398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" event={"ID":"234c1360-4185-4f41-b9a7-c2b09a729691","Type":"ContainerDied","Data":"317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e"} Sep 16 19:33:15.435623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.435423 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317cfbb6f847ec8e2039763c8dc6c84ef16712bf4fb8a5a64ba17f62dc55ed7e" Sep 16 19:33:15.447138 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.447116 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:33:15.500607 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.500575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs\") pod \"234c1360-4185-4f41-b9a7-c2b09a729691\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " Sep 16 19:33:15.500819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.500644 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgsth\" (UniqueName: \"kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth\") pod \"234c1360-4185-4f41-b9a7-c2b09a729691\" (UID: \"234c1360-4185-4f41-b9a7-c2b09a729691\") " Sep 16 19:33:15.502953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.502924 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "234c1360-4185-4f41-b9a7-c2b09a729691" (UID: "234c1360-4185-4f41-b9a7-c2b09a729691"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:33:15.503060 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.503024 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth" (OuterVolumeSpecName: "kube-api-access-dgsth") pod "234c1360-4185-4f41-b9a7-c2b09a729691" (UID: "234c1360-4185-4f41-b9a7-c2b09a729691"). InnerVolumeSpecName "kube-api-access-dgsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:33:15.601235 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.601196 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c1360-4185-4f41-b9a7-c2b09a729691-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:33:15.601235 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:15.601230 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsth\" (UniqueName: \"kubernetes.io/projected/234c1360-4185-4f41-b9a7-c2b09a729691-kube-api-access-dgsth\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:33:16.441949 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:16.441918 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m" Sep 16 19:33:16.460910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:16.460881 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:33:16.465636 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:16.465611 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-54cc44df67-zcd8m"] Sep 16 19:33:18.253956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:18.253912 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" path="/var/lib/kubelet/pods/234c1360-4185-4f41-b9a7-c2b09a729691/volumes" Sep 16 19:33:27.176834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.176800 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:33:27.177246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177229 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" Sep 16 19:33:27.177246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177240 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" Sep 16 19:33:27.177370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177249 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" Sep 16 19:33:27.177370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177254 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" Sep 16 19:33:27.177370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177333 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c42621e-6a57-482d-8e5b-9ce8fa84784a" containerName="main" Sep 16 19:33:27.177370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.177346 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="234c1360-4185-4f41-b9a7-c2b09a729691" containerName="main" Sep 16 19:33:27.180482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.180466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.183539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.183502 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\"" Sep 16 19:33:27.183688 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.183556 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\"" Sep 16 19:33:27.184633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.184610 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-kwrxz\"" Sep 16 19:33:27.184633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.184625 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\"" Sep 16 19:33:27.196509 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.196488 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:33:27.303936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.303889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.303936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.303942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8bb\" (UniqueName: \"kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.304191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.304029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.304191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.304084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.304191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.304105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.304191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.304175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405090 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8bb\" (UniqueName: \"kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405554 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405600 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405562 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.405600 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.405589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.407726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.407702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.407919 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.407904 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.416557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.416531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8bb\" (UniqueName: \"kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb\") pod \"canary-v1-kserve-65d58ccc9-lrkhm\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.492105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.492077 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:27.776710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.776680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:33:27.778236 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:33:27.778207 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c60073_355d_401d_9679_e6282bd0b851.slice/crio-c4e047e0bea4e4adbe1ca31e8a272f36742c907dc0b41d8a9759eff720c87a94 WatchSource:0}: Error finding container c4e047e0bea4e4adbe1ca31e8a272f36742c907dc0b41d8a9759eff720c87a94: Status 404 returned error can't find the container with id c4e047e0bea4e4adbe1ca31e8a272f36742c907dc0b41d8a9759eff720c87a94 Sep 16 19:33:27.780303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:27.780268 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:33:28.514834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:28.514788 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" event={"ID":"c0c60073-355d-401d-9679-e6282bd0b851","Type":"ContainerStarted","Data":"c4e047e0bea4e4adbe1ca31e8a272f36742c907dc0b41d8a9759eff720c87a94"} Sep 16 19:33:29.523039 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:29.523002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" event={"ID":"c0c60073-355d-401d-9679-e6282bd0b851","Type":"ContainerStarted","Data":"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac"} Sep 16 19:33:29.543444 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:29.543391 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" podStartSLOduration=1.561853178 podStartE2EDuration="2.543372207s" totalImagesPullingTime="981.519029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:29.542822603 +0000 UTC m=+1693.748778356" watchObservedRunningTime="2026-09-16 19:33:29.543372207 +0000 UTC m=+1693.749327961" Sep 16 19:33:30.493103 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:30.493065 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:37.493220 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:37.493126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:37.505481 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:37.505456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:37.578711 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:37.578682 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:33:53.988642 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:53.988601 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:33:53.992189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:53.992168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:53.994755 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:53.994724 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\"" Sep 16 19:33:54.016545 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.016505 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:33:54.054186 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.054422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.054422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.054422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.054422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.054422 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.054357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5qk\" (UniqueName: \"kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155368 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qk\" (UniqueName: \"kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.155885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.156042 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.155930 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.157807 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.157786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.158052 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.158036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.165202 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.165180 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5qk\" (UniqueName: \"kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk\") pod \"canary-v2-kserve-7f65dbbccc-r6fcv\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.302350 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.302228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:33:54.584659 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.584508 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:33:54.588136 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:33:54.588106 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f59a671_a839_482c_9bb7_dbdc2e5b732b.slice/crio-8b1b675a5d1501ff4ea56123d24fd65d01e711e41355746ee12442ca37fa95ca WatchSource:0}: Error finding container 8b1b675a5d1501ff4ea56123d24fd65d01e711e41355746ee12442ca37fa95ca: Status 404 returned error can't find the container with id 8b1b675a5d1501ff4ea56123d24fd65d01e711e41355746ee12442ca37fa95ca Sep 16 19:33:54.666637 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.666606 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" event={"ID":"0f59a671-a839-482c-9bb7-dbdc2e5b732b","Type":"ContainerStarted","Data":"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f"} Sep 16 19:33:54.666782 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.666666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" event={"ID":"0f59a671-a839-482c-9bb7-dbdc2e5b732b","Type":"ContainerStarted","Data":"8b1b675a5d1501ff4ea56123d24fd65d01e711e41355746ee12442ca37fa95ca"} Sep 16 19:33:54.693399 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:54.693339 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" podStartSLOduration=1.6933197789999999 podStartE2EDuration="1.693319779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:54.691519943 +0000 UTC m=+1718.897475697" watchObservedRunningTime="2026-09-16 19:33:54.693319779 +0000 UTC m=+1718.899275531" Sep 16 19:33:55.302693 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:33:55.302656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:34:04.303489 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:04.303455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:34:04.315977 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:04.315954 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:34:04.735311 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:04.735254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:34:56.789828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.789795 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:34:56.792368 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.792351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.799053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.798587 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\"" Sep 16 19:34:56.799053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.798651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-fkqtl\"" Sep 16 19:34:56.799053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.798689 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\"" Sep 16 19:34:56.799053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.798711 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-2bfvv\"" Sep 16 19:34:56.799053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.798651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\"" Sep 16 19:34:56.817304 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.816388 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:34:56.937824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266w8\" (UniqueName: \"kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.937824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.938045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.938045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.938045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:56.938045 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:56.937991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038459 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266w8\" (UniqueName: \"kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038459 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038467 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.038989 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.038966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.039048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.039012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.039048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.039036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.040990 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.040945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.041071 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.041053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.049108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.049087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-266w8\" (UniqueName: \"kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.102143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.102103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:34:57.389358 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:57.389318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:34:57.390970 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:34:57.390943 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28e4e44_b00a_4306_b48f_ab98204a2dce.slice/crio-2e7984dc3d223f4373a186561c531aef72f6bf5023a8170729b57f09c297bbf6 WatchSource:0}: Error finding container 2e7984dc3d223f4373a186561c531aef72f6bf5023a8170729b57f09c297bbf6: Status 404 returned error can't find the container with id 2e7984dc3d223f4373a186561c531aef72f6bf5023a8170729b57f09c297bbf6 Sep 16 19:34:58.040150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:58.040097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerStarted","Data":"2e7984dc3d223f4373a186561c531aef72f6bf5023a8170729b57f09c297bbf6"} Sep 16 19:34:59.048632 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:59.048598 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerStarted","Data":"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005"} Sep 16 19:34:59.048995 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:34:59.048671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:00.056830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:00.056785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerStarted","Data":"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f"} Sep 16 19:35:00.089832 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:00.089777 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" podStartSLOduration=2.6663044339999997 podStartE2EDuration="4.08975954s" totalImagesPullingTime="1.423455106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:00.085987031 +0000 UTC m=+1784.291942781" watchObservedRunningTime="2026-09-16 19:35:00.08975954 +0000 UTC m=+1784.295715294" Sep 16 19:35:00.103184 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:00.103149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:07.102457 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:07.102427 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:07.114849 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:07.114821 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:08.114041 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:08.114010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:08.119861 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:08.119840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:32.090978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.090943 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:35:32.101690 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:32.101638 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:35:32.101690 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:35:32.101690 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:35:32.101690 ip-10-0-128-148 kubenswrapper[2565]: > containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" cmd=["/bin/sleep","15"] Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.101713 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" message="" err=< Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:32.101731 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" containerID="cri-o://74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" Sep 16 19:35:32.101936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.101796 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" containerID="cri-o://74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" gracePeriod=60 Sep 16 19:35:32.114786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.114756 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8001/health\": dial tcp 10.133.0.67:8001: connect: connection refused" Sep 16 19:35:32.320237 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.320205 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="llm-d-routing-sidecar" containerID="cri-o://e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" gracePeriod=60 Sep 16 19:35:32.573183 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.573155 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:32.664051 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664012 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664101 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-266w8\" (UniqueName: \"kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664429 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664252 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache\") pod \"e28e4e44-b00a-4306-b48f-ab98204a2dce\" (UID: \"e28e4e44-b00a-4306-b48f-ab98204a2dce\") " Sep 16 19:35:32.664429 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home" (OuterVolumeSpecName: "home") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:32.664541 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664439 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:32.664593 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache" (OuterVolumeSpecName: "model-cache") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:32.664650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664617 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:32.664650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664636 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:32.664650 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.664646 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:32.666588 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.666563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm" (OuterVolumeSpecName: "dshm") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:32.666696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.666604 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:35:32.666696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.666620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8" (OuterVolumeSpecName: "kube-api-access-266w8") pod "e28e4e44-b00a-4306-b48f-ab98204a2dce" (UID: "e28e4e44-b00a-4306-b48f-ab98204a2dce"). InnerVolumeSpecName "kube-api-access-266w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:35:32.766082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.766044 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e28e4e44-b00a-4306-b48f-ab98204a2dce-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:32.766082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.766076 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28e4e44-b00a-4306-b48f-ab98204a2dce-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:32.766082 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:32.766087 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266w8\" (UniqueName: \"kubernetes.io/projected/e28e4e44-b00a-4306-b48f-ab98204a2dce-kube-api-access-266w8\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:33.264095 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264060 2565 generic.go:356] "Generic (PLEG): container finished" podID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" exitCode=0 Sep 16 19:35:33.264095 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264085 2565 generic.go:356] "Generic (PLEG): container finished" podID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerID="e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" exitCode=0 Sep 16 19:35:33.264609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264117 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" Sep 16 19:35:33.264609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerDied","Data":"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f"} Sep 16 19:35:33.264609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerDied","Data":"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005"} Sep 16 19:35:33.264609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2" event={"ID":"e28e4e44-b00a-4306-b48f-ab98204a2dce","Type":"ContainerDied","Data":"2e7984dc3d223f4373a186561c531aef72f6bf5023a8170729b57f09c297bbf6"} Sep 16 19:35:33.264609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.264264 2565 scope.go:117] "RemoveContainer" containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" Sep 16 19:35:33.275000 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.274980 2565 scope.go:117] "RemoveContainer" containerID="e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" Sep 16 19:35:33.284423 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.284401 2565 scope.go:117] "RemoveContainer" containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" Sep 16 19:35:33.284719 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:33.284699 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f\": container with ID starting with 74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f not found: ID does not exist" containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" Sep 16 19:35:33.284775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.284730 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f"} err="failed to get container status \"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f\": rpc error: code = NotFound desc = could not find container \"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f\": container with ID starting with 74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f not found: ID does not exist" Sep 16 19:35:33.284775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.284750 2565 scope.go:117] "RemoveContainer" containerID="e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" Sep 16 19:35:33.285016 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:33.284999 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005\": container with ID starting with e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005 not found: ID does not exist" containerID="e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" Sep 16 19:35:33.285077 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.285019 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005"} err="failed to get container status \"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005\": rpc error: code = NotFound desc = could not find container \"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005\": container with ID starting with e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005 not found: ID does not exist" Sep 16 19:35:33.285077 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.285033 2565 scope.go:117] "RemoveContainer" containerID="74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f" Sep 16 19:35:33.285307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.285262 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f"} err="failed to get container status \"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f\": rpc error: code = NotFound desc = could not find container \"74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f\": container with ID starting with 74b3440c818c1a9ec0971ce2592acb81ce0014c2e818becb2e496783b3ce8d7f not found: ID does not exist" Sep 16 19:35:33.285361 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.285348 2565 scope.go:117] "RemoveContainer" containerID="e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005" Sep 16 19:35:33.285646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.285624 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005"} err="failed to get container status \"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005\": rpc error: code = NotFound desc = could not find container \"e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005\": container with ID starting with e55f24597aee76cc2f6233b88f947b930111cbdf58f82f3499020f3a078f9005 not found: ID does not exist" Sep 16 19:35:33.290220 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.290197 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:35:33.300182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:33.300159 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-tzws2"] Sep 16 19:35:34.253833 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:34.253795 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" path="/var/lib/kubelet/pods/e28e4e44-b00a-4306-b48f-ab98204a2dce/volumes" Sep 16 19:35:42.843315 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:42.843242 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:35:43.070882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:43.070850 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:35:57.864458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:57.864413 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" podUID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" containerName="main" containerID="cri-o://40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f" gracePeriod=45 Sep 16 19:35:58.094238 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.094193 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" podUID="c0c60073-355d-401d-9679-e6282bd0b851" containerName="main" containerID="cri-o://f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac" gracePeriod=45 Sep 16 19:35:58.220361 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.220334 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:35:58.313024 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.312991 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313052 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313081 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb5qk\" (UniqueName: \"kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313118 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313138 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313181 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs\") pod \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\" (UID: \"0f59a671-a839-482c-9bb7-dbdc2e5b732b\") " Sep 16 19:35:58.313521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313420 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.313521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.313521 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313468 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home" (OuterVolumeSpecName: "home") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.313680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313542 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.313680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313554 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.313680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.313565 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.316331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.316297 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm" (OuterVolumeSpecName: "dshm") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.316465 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.316378 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:35:58.316465 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.316386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk" (OuterVolumeSpecName: "kube-api-access-fb5qk") pod "0f59a671-a839-482c-9bb7-dbdc2e5b732b" (UID: "0f59a671-a839-482c-9bb7-dbdc2e5b732b"). InnerVolumeSpecName "kube-api-access-fb5qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:35:58.342836 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.342813 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:35:58.411272 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.411183 2565 generic.go:356] "Generic (PLEG): container finished" podID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" containerID="40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f" exitCode=0 Sep 16 19:35:58.411272 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.411238 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" Sep 16 19:35:58.411272 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.411273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" event={"ID":"0f59a671-a839-482c-9bb7-dbdc2e5b732b","Type":"ContainerDied","Data":"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f"} Sep 16 19:35:58.411571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.411339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv" event={"ID":"0f59a671-a839-482c-9bb7-dbdc2e5b732b","Type":"ContainerDied","Data":"8b1b675a5d1501ff4ea56123d24fd65d01e711e41355746ee12442ca37fa95ca"} Sep 16 19:35:58.411571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.411365 2565 scope.go:117] "RemoveContainer" containerID="40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f" Sep 16 19:35:58.412667 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.412646 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0c60073-355d-401d-9679-e6282bd0b851" containerID="f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac" exitCode=0 Sep 16 19:35:58.412760 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.412690 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" Sep 16 19:35:58.412760 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.412680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" event={"ID":"c0c60073-355d-401d-9679-e6282bd0b851","Type":"ContainerDied","Data":"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac"} Sep 16 19:35:58.412760 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.412740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm" event={"ID":"c0c60073-355d-401d-9679-e6282bd0b851","Type":"ContainerDied","Data":"c4e047e0bea4e4adbe1ca31e8a272f36742c907dc0b41d8a9759eff720c87a94"} Sep 16 19:35:58.414187 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.414165 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb5qk\" (UniqueName: \"kubernetes.io/projected/0f59a671-a839-482c-9bb7-dbdc2e5b732b-kube-api-access-fb5qk\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.414187 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.414186 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f59a671-a839-482c-9bb7-dbdc2e5b732b-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.414380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.414195 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f59a671-a839-482c-9bb7-dbdc2e5b732b-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.421527 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.421510 2565 scope.go:117] "RemoveContainer" containerID="40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f" Sep 16 19:35:58.421801 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:58.421783 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f\": container with ID starting with 40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f not found: ID does not exist" containerID="40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f" Sep 16 19:35:58.421871 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.421809 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f"} err="failed to get container status \"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f\": rpc error: code = NotFound desc = could not find container \"40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f\": container with ID starting with 40bac89faeabe552600d0d568bd3f5dd98d515e3bbe87b29e7cd73065a681b8f not found: ID does not exist" Sep 16 19:35:58.421871 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.421824 2565 scope.go:117] "RemoveContainer" containerID="f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac" Sep 16 19:35:58.429936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.429915 2565 scope.go:117] "RemoveContainer" containerID="f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac" Sep 16 19:35:58.430219 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:35:58.430200 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac\": container with ID starting with f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac not found: ID does not exist" containerID="f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac" Sep 16 19:35:58.430266 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.430225 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac"} err="failed to get container status \"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac\": rpc error: code = NotFound desc = could not find container \"f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac\": container with ID starting with f6543487b581b12b7f109ab6fcf57fafce853ecbb5035f19dc390a0c912dd9ac not found: ID does not exist" Sep 16 19:35:58.440168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.440144 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:35:58.446687 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.446663 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-r6fcv"] Sep 16 19:35:58.515472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515438 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515526 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p8bb\" (UniqueName: \"kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515584 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir\") pod \"c0c60073-355d-401d-9679-e6282bd0b851\" (UID: \"c0c60073-355d-401d-9679-e6282bd0b851\") " Sep 16 19:35:58.515907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515750 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home" (OuterVolumeSpecName: "home") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.515907 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515865 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.516020 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.515927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.516058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.516042 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.516097 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.516060 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.516097 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.516073 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.517823 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.517799 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm" (OuterVolumeSpecName: "dshm") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:35:58.517889 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.517862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:35:58.517926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.517886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb" (OuterVolumeSpecName: "kube-api-access-8p8bb") pod "c0c60073-355d-401d-9679-e6282bd0b851" (UID: "c0c60073-355d-401d-9679-e6282bd0b851"). InnerVolumeSpecName "kube-api-access-8p8bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:35:58.617419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.617383 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0c60073-355d-401d-9679-e6282bd0b851-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.617419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.617412 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p8bb\" (UniqueName: \"kubernetes.io/projected/c0c60073-355d-401d-9679-e6282bd0b851-kube-api-access-8p8bb\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.617419 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.617425 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0c60073-355d-401d-9679-e6282bd0b851-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:35:58.740763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.740723 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:35:58.743801 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:35:58.743779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-lrkhm"] Sep 16 19:36:00.253889 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:00.253854 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" path="/var/lib/kubelet/pods/0f59a671-a839-482c-9bb7-dbdc2e5b732b/volumes" Sep 16 19:36:00.254268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:00.254242 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c60073-355d-401d-9679-e6282bd0b851" path="/var/lib/kubelet/pods/c0c60073-355d-401d-9679-e6282bd0b851/volumes" Sep 16 19:36:02.966902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.966865 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:36:02.967306 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967253 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="llm-d-routing-sidecar" Sep 16 19:36:02.967306 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967265 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="llm-d-routing-sidecar" Sep 16 19:36:02.967306 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967298 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c60073-355d-401d-9679-e6282bd0b851" containerName="main" Sep 16 19:36:02.967306 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967304 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c60073-355d-401d-9679-e6282bd0b851" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967316 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967323 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967335 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967340 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967393 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="llm-d-routing-sidecar" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967402 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c60073-355d-401d-9679-e6282bd0b851" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967409 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f59a671-a839-482c-9bb7-dbdc2e5b732b" containerName="main" Sep 16 19:36:02.967448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.967413 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28e4e44-b00a-4306-b48f-ab98204a2dce" containerName="main" Sep 16 19:36:02.972127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.972108 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:02.975495 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.975466 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\"" Sep 16 19:36:02.976271 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.976249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-c7bqg\"" Sep 16 19:36:02.976271 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.976271 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\"" Sep 16 19:36:02.976457 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.976391 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\"" Sep 16 19:36:02.988672 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:02.988647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:36:03.057368 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4nq\" (UniqueName: \"kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.057550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.057550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.057550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.057550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.057762 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.057658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158074 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158345 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158345 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4nq\" (UniqueName: \"kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158345 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158345 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158630 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.158691 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.158639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.160582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.160564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.160852 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.160832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.167433 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.167411 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4nq\" (UniqueName: \"kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq\") pod \"test-div-v1-kserve-fb566d667-ls4cv\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.282698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.282590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:03.585598 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:36:03.585562 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4b7875_4428_4b3f_a36a_f03400f4d480.slice/crio-521da77e87ca789855f739a0d2f7cb903cb6f74af39053c0bb44259d893e6d00 WatchSource:0}: Error finding container 521da77e87ca789855f739a0d2f7cb903cb6f74af39053c0bb44259d893e6d00: Status 404 returned error can't find the container with id 521da77e87ca789855f739a0d2f7cb903cb6f74af39053c0bb44259d893e6d00 Sep 16 19:36:03.585733 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:03.585709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:36:04.452079 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:04.452040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" event={"ID":"7c4b7875-4428-4b3f-a36a-f03400f4d480","Type":"ContainerStarted","Data":"338ddc2f7ea44c764c8d02f30e6cbf8077dcc479a5601f01c1d3f4e375ec611b"} Sep 16 19:36:04.452497 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:04.452098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" event={"ID":"7c4b7875-4428-4b3f-a36a-f03400f4d480","Type":"ContainerStarted","Data":"521da77e87ca789855f739a0d2f7cb903cb6f74af39053c0bb44259d893e6d00"} Sep 16 19:36:04.489159 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:04.489100 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podStartSLOduration=2.489081077 podStartE2EDuration="2.489081077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:04.488554524 +0000 UTC m=+1848.694510281" watchObservedRunningTime="2026-09-16 19:36:04.489081077 +0000 UTC m=+1848.695036829" Sep 16 19:36:05.283711 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:05.283674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:13.282834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:13.282789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:13.295738 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:13.295708 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:13.515503 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:13.515473 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:36:14.038993 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.038962 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:36:14.042830 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.042803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.048023 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.048002 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\"" Sep 16 19:36:14.061651 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.061616 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:36:14.158555 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.158555 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.158824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.158824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.158824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.158824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.158746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwpm\" (UniqueName: \"kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259379 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259572 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259643 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259643 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259643 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwpm\" (UniqueName: \"kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.259977 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.259958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.262183 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.262159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.262313 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.262296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.268622 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.268600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwpm\" (UniqueName: \"kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm\") pod \"test-div-v2-kserve-5f6fdb6b77-pnxrh\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.353658 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.353567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:14.656081 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:14.656047 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:36:14.657878 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:36:14.657851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9774cb6f_6cce_4cc0_a1d1_e76b9320f152.slice/crio-8e12e7ce8a24d901e88eb896f39846ba6fb4c92a04c12b69bffe27c528393bdc WatchSource:0}: Error finding container 8e12e7ce8a24d901e88eb896f39846ba6fb4c92a04c12b69bffe27c528393bdc: Status 404 returned error can't find the container with id 8e12e7ce8a24d901e88eb896f39846ba6fb4c92a04c12b69bffe27c528393bdc Sep 16 19:36:15.520302 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:15.520239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" event={"ID":"9774cb6f-6cce-4cc0-a1d1-e76b9320f152","Type":"ContainerStarted","Data":"778459bfcce82a4ca3456fa84644551a5d677ae0c7c6d45633bdc8b0fe3a1943"} Sep 16 19:36:15.520302 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:15.520310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" event={"ID":"9774cb6f-6cce-4cc0-a1d1-e76b9320f152","Type":"ContainerStarted","Data":"8e12e7ce8a24d901e88eb896f39846ba6fb4c92a04c12b69bffe27c528393bdc"} Sep 16 19:36:15.550486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:15.550430 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podStartSLOduration=1.550414849 podStartE2EDuration="1.550414849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:15.550053735 +0000 UTC m=+1859.756009489" watchObservedRunningTime="2026-09-16 19:36:15.550414849 +0000 UTC m=+1859.756370602" Sep 16 19:36:16.354630 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:16.354592 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:24.354495 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:24.354454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:24.367420 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:24.367385 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:24.582580 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:24.582552 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:36:41.250570 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.250482 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:36:41.255819 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.255790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.265321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.265299 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\"" Sep 16 19:36:41.265777 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.265760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\"" Sep 16 19:36:41.266457 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.266443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\"" Sep 16 19:36:41.266535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.266521 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\"" Sep 16 19:36:41.266832 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.266819 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:36:41.267659 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.267645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-4vqpk\"" Sep 16 19:36:41.307293 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.307248 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:36:41.413168 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjtr\" (UniqueName: \"kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413387 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.413601 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.413590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjtr\" (UniqueName: \"kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.514901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.515174 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.514995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.515174 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.515155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.515313 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.515196 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.515313 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.515244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.515629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.515606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.517257 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.517235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.517606 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.517589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.526314 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.526268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjtr\" (UniqueName: \"kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr\") pod \"tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.563137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.563105 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:36:41.567568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.567540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:41.568024 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.568006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.579613 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.579587 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:36:41.716157 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.716111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.716396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.716178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.716396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.716235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.716396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.716346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.817903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.817817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.817903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.817881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.818146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.817934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.818146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.817985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.818439 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.818413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.818537 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.818451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.818789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.818769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.820740 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.820715 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:41.898112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.898086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:36:41.900030 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:36:41.900005 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27161b7d_98cc_4249_8141_c99c86e2ea7d.slice/crio-435b1eced582629e0382191615c8873cbb00494157234a71ba775944dceb81cd WatchSource:0}: Error finding container 435b1eced582629e0382191615c8873cbb00494157234a71ba775944dceb81cd: Status 404 returned error can't find the container with id 435b1eced582629e0382191615c8873cbb00494157234a71ba775944dceb81cd Sep 16 19:36:41.901367 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:41.901350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:36:42.189800 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:42.189771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:36:42.190705 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:36:42.190681 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd6b55e_57cb_4e51_a55a_cfdea8de46f4.slice/crio-581359b8d3953275ec0a89d03bf46a845d2584d86ed2c83114b4ccff88c817b1 WatchSource:0}: Error finding container 581359b8d3953275ec0a89d03bf46a845d2584d86ed2c83114b4ccff88c817b1: Status 404 returned error can't find the container with id 581359b8d3953275ec0a89d03bf46a845d2584d86ed2c83114b4ccff88c817b1 Sep 16 19:36:42.677189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:42.677149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerStarted","Data":"fc25100e467ee2694868809a838fa283fd354f3f0b7fb72e1b0865094f5f8817"} Sep 16 19:36:42.677696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:42.677209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerStarted","Data":"435b1eced582629e0382191615c8873cbb00494157234a71ba775944dceb81cd"} Sep 16 19:36:42.679057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:42.679031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerStarted","Data":"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe"} Sep 16 19:36:42.679188 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:42.679090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerStarted","Data":"581359b8d3953275ec0a89d03bf46a845d2584d86ed2c83114b4ccff88c817b1"} Sep 16 19:36:44.698466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:44.698423 2565 generic.go:356] "Generic (PLEG): container finished" podID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerID="fc25100e467ee2694868809a838fa283fd354f3f0b7fb72e1b0865094f5f8817" exitCode=0 Sep 16 19:36:44.698901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:44.698516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerDied","Data":"fc25100e467ee2694868809a838fa283fd354f3f0b7fb72e1b0865094f5f8817"} Sep 16 19:36:45.707675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:45.707633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerStarted","Data":"815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6"} Sep 16 19:36:45.709389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:45.709362 2565 generic.go:356] "Generic (PLEG): container finished" podID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerID="3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe" exitCode=0 Sep 16 19:36:45.709529 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:45.709414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerDied","Data":"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe"} Sep 16 19:36:45.734014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:45.733958 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" podStartSLOduration=2.146544776 podStartE2EDuration="4.733937313s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.587392537s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:45.731771898 +0000 UTC m=+1889.937727652" watchObservedRunningTime="2026-09-16 19:36:45.733937313 +0000 UTC m=+1889.939893066" Sep 16 19:36:46.568298 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:46.568246 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:51.568396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:51.568325 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:51.588153 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:51.587892 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:36:51.775250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:36:51.775218 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:37:02.669571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:02.669521 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:37:04.039353 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:04.039243 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:37:16.537185 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:16.537148 2565 scope.go:117] "RemoveContainer" containerID="f2e67032d53de56a9e9a415df62e160865c75a005a94b6f8374bcf6d28b0edfb" Sep 16 19:37:17.701371 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:17.701333 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" containerID="cri-o://338ddc2f7ea44c764c8d02f30e6cbf8077dcc479a5601f01c1d3f4e375ec611b" gracePeriod=45 Sep 16 19:37:18.505111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:18.505059 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:19.067384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:19.067252 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" containerID="cri-o://778459bfcce82a4ca3456fa84644551a5d677ae0c7c6d45633bdc8b0fe3a1943" gracePeriod=45 Sep 16 19:37:19.505501 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:19.505452 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:19.505681 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:19.505595 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:37:19.572640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:19.572587 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:20.504719 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:20.504652 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:20.572298 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:20.572216 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:20.572485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:20.572392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:37:21.505659 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:21.505598 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:21.571992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:21.571947 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:22.505551 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:22.505495 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:22.572272 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:22.572226 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:23.505568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:23.505510 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:23.572094 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:23.572054 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:24.504649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:24.504595 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:24.572377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:24.572332 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:25.505244 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:25.505187 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:25.571632 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:25.571574 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:26.505372 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:26.505320 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:26.572409 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:26.572371 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:27.505025 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:27.504972 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:27.571740 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:27.571695 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:28.505037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:28.504995 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:28.572220 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:28.572177 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:29.504718 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:29.504661 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:29.572618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:29.572571 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:30.505624 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:30.505571 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:30.572541 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:30.572482 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:31.504990 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:31.504941 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:31.572271 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:31.572227 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:32.504712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:32.504662 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused" Sep 16 19:37:32.572190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:32.572150 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused" Sep 16 19:37:33.081091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.081054 2565 generic.go:356] "Generic (PLEG): container finished" podID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerID="778459bfcce82a4ca3456fa84644551a5d677ae0c7c6d45633bdc8b0fe3a1943" exitCode=0 Sep 16 19:37:33.081341 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.081302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" event={"ID":"9774cb6f-6cce-4cc0-a1d1-e76b9320f152","Type":"ContainerDied","Data":"778459bfcce82a4ca3456fa84644551a5d677ae0c7c6d45633bdc8b0fe3a1943"} Sep 16 19:37:33.083394 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.083367 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerID="338ddc2f7ea44c764c8d02f30e6cbf8077dcc479a5601f01c1d3f4e375ec611b" exitCode=0 Sep 16 19:37:33.083533 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.083440 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" event={"ID":"7c4b7875-4428-4b3f-a36a-f03400f4d480","Type":"ContainerDied","Data":"338ddc2f7ea44c764c8d02f30e6cbf8077dcc479a5601f01c1d3f4e375ec611b"} Sep 16 19:37:33.148628 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.148596 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:37:33.152337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.152314 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:37:33.249726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249690 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.249901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249750 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.249901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249773 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.249901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249845 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.249901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249880 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4nq\" (UniqueName: \"kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249938 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249953 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249964 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwpm\" (UniqueName: \"kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.249999 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache" (OuterVolumeSpecName: "model-cache") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250026 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250052 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250081 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home\") pod \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\" (UID: \"9774cb6f-6cce-4cc0-a1d1-e76b9320f152\") " Sep 16 19:37:33.250127 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache\") pod \"7c4b7875-4428-4b3f-a36a-f03400f4d480\" (UID: \"7c4b7875-4428-4b3f-a36a-f03400f4d480\") " Sep 16 19:37:33.250822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250475 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.250822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250489 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.250822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250509 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.250822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home" (OuterVolumeSpecName: "home") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.250822 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.250730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home" (OuterVolumeSpecName: "home") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.251192 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.251120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.253013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.252981 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:37:33.253013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.252994 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq" (OuterVolumeSpecName: "kube-api-access-7s4nq") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "kube-api-access-7s4nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:37:33.253013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.252989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:37:33.253241 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.252988 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm" (OuterVolumeSpecName: "dshm") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.253241 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.253054 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm" (OuterVolumeSpecName: "kube-api-access-xvwpm") pod "9774cb6f-6cce-4cc0-a1d1-e76b9320f152" (UID: "9774cb6f-6cce-4cc0-a1d1-e76b9320f152"). InnerVolumeSpecName "kube-api-access-xvwpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:37:33.253386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.253241 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm" (OuterVolumeSpecName: "dshm") pod "7c4b7875-4428-4b3f-a36a-f03400f4d480" (UID: "7c4b7875-4428-4b3f-a36a-f03400f4d480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:33.351423 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351389 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4b7875-4428-4b3f-a36a-f03400f4d480-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351423 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351421 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4nq\" (UniqueName: \"kubernetes.io/projected/7c4b7875-4428-4b3f-a36a-f03400f4d480-kube-api-access-7s4nq\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351437 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351450 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351463 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwpm\" (UniqueName: \"kubernetes.io/projected/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-kube-api-access-xvwpm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351476 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351489 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351501 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351513 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7875-4428-4b3f-a36a-f03400f4d480-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:33.351629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:33.351525 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774cb6f-6cce-4cc0-a1d1-e76b9320f152-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:37:34.092599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.092568 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" Sep 16 19:37:34.092599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.092588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh" event={"ID":"9774cb6f-6cce-4cc0-a1d1-e76b9320f152","Type":"ContainerDied","Data":"8e12e7ce8a24d901e88eb896f39846ba6fb4c92a04c12b69bffe27c528393bdc"} Sep 16 19:37:34.093234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.092652 2565 scope.go:117] "RemoveContainer" containerID="778459bfcce82a4ca3456fa84644551a5d677ae0c7c6d45633bdc8b0fe3a1943" Sep 16 19:37:34.094233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.094208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" event={"ID":"7c4b7875-4428-4b3f-a36a-f03400f4d480","Type":"ContainerDied","Data":"521da77e87ca789855f739a0d2f7cb903cb6f74af39053c0bb44259d893e6d00"} Sep 16 19:37:34.094233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.094231 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv" Sep 16 19:37:34.096726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.096699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerStarted","Data":"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f"} Sep 16 19:37:34.097018 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.096999 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:37:34.104349 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.104327 2565 scope.go:117] "RemoveContainer" containerID="338ddc2f7ea44c764c8d02f30e6cbf8077dcc479a5601f01c1d3f4e375ec611b" Sep 16 19:37:34.124900 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.124844 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" podStartSLOduration=2.945054063 podStartE2EDuration="53.124822491s" totalImagesPullingTime="47.562046397s" totalInitContainerRuntime="2.617722031s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:34.122667813 +0000 UTC m=+1938.328623574" watchObservedRunningTime="2026-09-16 19:37:34.124822491 +0000 UTC m=+1938.330778245" Sep 16 19:37:34.145680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.145649 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:37:34.149786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.149757 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-pnxrh"] Sep 16 19:37:34.158142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.158113 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:37:34.162978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.162957 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-ls4cv"] Sep 16 19:37:34.254466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.254425 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" path="/var/lib/kubelet/pods/7c4b7875-4428-4b3f-a36a-f03400f4d480/volumes" Sep 16 19:37:34.255100 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:37:34.255076 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" path="/var/lib/kubelet/pods/9774cb6f-6cce-4cc0-a1d1-e76b9320f152/volumes" Sep 16 19:38:05.110081 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:05.110032 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused" Sep 16 19:38:15.117759 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:15.117664 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:38:40.492932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:40.492903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5_edd6b55e-57cb-4e51-a55a-cfdea8de46f4/main/0.log" Sep 16 19:38:48.254741 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.254712 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:38:48.262515 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.262483 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:38:48.262803 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.262742 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="main" containerID="cri-o://ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f" gracePeriod=30 Sep 16 19:38:48.266075 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:38:48.266038 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:38:48.266075 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:38:48.266075 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:38:48.266075 ip-10-0-128-148 kubenswrapper[2565]: > containerID="815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6" cmd=["/bin/sleep","15"] Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.266095 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" message="" err=< Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:38:48.266113 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="main" containerID="cri-o://815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6" Sep 16 19:38:48.266414 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.266165 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="main" containerID="cri-o://815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6" gracePeriod=60 Sep 16 19:38:48.548016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.547966 2565 generic.go:356] "Generic (PLEG): container finished" podID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerID="815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6" exitCode=0 Sep 16 19:38:48.548372 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.548334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerDied","Data":"815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6"} Sep 16 19:38:48.627834 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.627806 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:38:48.770912 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.770870 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.770932 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.770963 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.770996 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgjtr\" (UniqueName: \"kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771021 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771110 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771411 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771136 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771411 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771174 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache\") pod \"27161b7d-98cc-4249-8141-c99c86e2ea7d\" (UID: \"27161b7d-98cc-4249-8141-c99c86e2ea7d\") " Sep 16 19:38:48.771808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.772138 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771813 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.772138 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771883 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home" (OuterVolumeSpecName: "home") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.772138 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.771927 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:38:48.773999 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.773954 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:38:48.774187 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.774082 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm" (OuterVolumeSpecName: "dshm") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.776449 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.775694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr" (OuterVolumeSpecName: "kube-api-access-lgjtr") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "kube-api-access-lgjtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:38:48.784122 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.784088 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27161b7d-98cc-4249-8141-c99c86e2ea7d" (UID: "27161b7d-98cc-4249-8141-c99c86e2ea7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.836080 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.836049 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:38:48.872389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872355 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27161b7d-98cc-4249-8141-c99c86e2ea7d-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872391 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872405 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872417 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872430 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872443 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjtr\" (UniqueName: \"kubernetes.io/projected/27161b7d-98cc-4249-8141-c99c86e2ea7d-kube-api-access-lgjtr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872457 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27161b7d-98cc-4249-8141-c99c86e2ea7d-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.872586 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.872469 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27161b7d-98cc-4249-8141-c99c86e2ea7d-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:48.973393 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973359 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location\") pod \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " Sep 16 19:38:48.973571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs\") pod \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " Sep 16 19:38:48.973571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert\") pod \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " Sep 16 19:38:48.973571 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir\") pod \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\" (UID: \"edd6b55e-57cb-4e51-a55a-cfdea8de46f4\") " Sep 16 19:38:48.973868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973843 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edd6b55e-57cb-4e51-a55a-cfdea8de46f4" (UID: "edd6b55e-57cb-4e51-a55a-cfdea8de46f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:38:48.973988 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.973964 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edd6b55e-57cb-4e51-a55a-cfdea8de46f4" (UID: "edd6b55e-57cb-4e51-a55a-cfdea8de46f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:48.975808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.975786 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edd6b55e-57cb-4e51-a55a-cfdea8de46f4" (UID: "edd6b55e-57cb-4e51-a55a-cfdea8de46f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:38:48.980361 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:48.980339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edd6b55e-57cb-4e51-a55a-cfdea8de46f4" (UID: "edd6b55e-57cb-4e51-a55a-cfdea8de46f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:38:49.074499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.074464 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:49.074499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.074494 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:49.074499 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.074504 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:49.074746 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.074512 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edd6b55e-57cb-4e51-a55a-cfdea8de46f4-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:38:49.555347 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.555315 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" Sep 16 19:38:49.555347 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.555325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg" event={"ID":"27161b7d-98cc-4249-8141-c99c86e2ea7d","Type":"ContainerDied","Data":"435b1eced582629e0382191615c8873cbb00494157234a71ba775944dceb81cd"} Sep 16 19:38:49.555862 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.555385 2565 scope.go:117] "RemoveContainer" containerID="815e409043f5ca2771bbc89895e8b0d41f3c11ae8ed31c8fb40b01515f3ad2b6" Sep 16 19:38:49.557178 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.557154 2565 generic.go:356] "Generic (PLEG): container finished" podID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerID="ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f" exitCode=0 Sep 16 19:38:49.557276 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.557201 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" Sep 16 19:38:49.557276 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.557235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerDied","Data":"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f"} Sep 16 19:38:49.557392 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.557354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5" event={"ID":"edd6b55e-57cb-4e51-a55a-cfdea8de46f4","Type":"ContainerDied","Data":"581359b8d3953275ec0a89d03bf46a845d2584d86ed2c83114b4ccff88c817b1"} Sep 16 19:38:49.566386 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.566371 2565 scope.go:117] "RemoveContainer" containerID="fc25100e467ee2694868809a838fa283fd354f3f0b7fb72e1b0865094f5f8817" Sep 16 19:38:49.574967 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.574949 2565 scope.go:117] "RemoveContainer" containerID="ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f" Sep 16 19:38:49.584429 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.584407 2565 scope.go:117] "RemoveContainer" containerID="3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe" Sep 16 19:38:49.592824 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.592805 2565 scope.go:117] "RemoveContainer" containerID="ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f" Sep 16 19:38:49.593111 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:38:49.593091 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f\": container with ID starting with ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f not found: ID does not exist" containerID="ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f" Sep 16 19:38:49.593171 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.593120 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f"} err="failed to get container status \"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f\": rpc error: code = NotFound desc = could not find container \"ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f\": container with ID starting with ed21ca4ee2c5b081713db21106c75f4e45ac8de356c8436a3b341aafc6b7472f not found: ID does not exist" Sep 16 19:38:49.593171 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.593139 2565 scope.go:117] "RemoveContainer" containerID="3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe" Sep 16 19:38:49.593428 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:38:49.593411 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe\": container with ID starting with 3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe not found: ID does not exist" containerID="3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe" Sep 16 19:38:49.593483 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.593433 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe"} err="failed to get container status \"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe\": rpc error: code = NotFound desc = could not find container \"3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe\": container with ID starting with 3389bbfcfaaf0d69e2fd1fe06eca13fa233db4472ad2a23a4822023eb35023fe not found: ID does not exist" Sep 16 19:38:49.605058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.605035 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:38:49.610239 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.610218 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6d86d8df4b-6p5wg"] Sep 16 19:38:49.650438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.650356 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:38:49.653426 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:49.653404 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6998f64d59-9dnt5"] Sep 16 19:38:50.253775 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:50.253739 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" path="/var/lib/kubelet/pods/27161b7d-98cc-4249-8141-c99c86e2ea7d/volumes" Sep 16 19:38:50.254190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:38:50.254175 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" path="/var/lib/kubelet/pods/edd6b55e-57cb-4e51-a55a-cfdea8de46f4/volumes" Sep 16 19:40:39.593099 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593059 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593481 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593497 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593518 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593524 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593533 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="storage-initializer" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593540 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="storage-initializer" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593548 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="storage-initializer" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593553 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="storage-initializer" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593560 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593565 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593577 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="main" Sep 16 19:40:39.593620 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593582 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="main" Sep 16 19:40:39.593992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593638 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4b7875-4428-4b3f-a36a-f03400f4d480" containerName="main" Sep 16 19:40:39.593992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593645 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774cb6f-6cce-4cc0-a1d1-e76b9320f152" containerName="main" Sep 16 19:40:39.593992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593651 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="27161b7d-98cc-4249-8141-c99c86e2ea7d" containerName="main" Sep 16 19:40:39.593992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.593658 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd6b55e-57cb-4e51-a55a-cfdea8de46f4" containerName="main" Sep 16 19:40:39.595860 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.595841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.599323 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.599277 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\"" Sep 16 19:40:39.599453 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.599353 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\"" Sep 16 19:40:39.599924 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.599910 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:40:39.600029 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.599910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\"" Sep 16 19:40:39.600539 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.600504 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-p5tq5\"" Sep 16 19:40:39.600625 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.600519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\"" Sep 16 19:40:39.611483 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.611460 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:40:39.664226 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664226 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29d66\" (UniqueName: \"kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.664480 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.664459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.765922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.765881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.765922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.765927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766407 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d66\" (UniqueName: \"kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766407 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766407 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766407 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766592 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766443 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766592 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.766710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.766602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.767042 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.767022 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.768826 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.768800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.769000 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.768982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.776264 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.776241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d66\" (UniqueName: \"kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66\") pod \"tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.879554 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.879474 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:40:39.883478 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.883455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:39.890976 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.890946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:40:39.906102 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.906066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:39.968999 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.968968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:39.969220 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.969193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:39.969392 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.969353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:39.969525 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:39.969401 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.070786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.070741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.070786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.070786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.071056 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.070914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.071056 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.070958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.071242 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.071220 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.071330 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.071295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.071593 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.071576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.073709 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.073690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.196181 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.196156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:40:40.197073 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.197051 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:40.197480 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:40:40.197453 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506f23ea_ac27_4142_a2db_ea49b874d727.slice/crio-903a63f2a57baadde5ca097b5511973fbd9debf14a9145927d3ca3d87a3bfb1a WatchSource:0}: Error finding container 903a63f2a57baadde5ca097b5511973fbd9debf14a9145927d3ca3d87a3bfb1a: Status 404 returned error can't find the container with id 903a63f2a57baadde5ca097b5511973fbd9debf14a9145927d3ca3d87a3bfb1a Sep 16 19:40:40.487444 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:40.487414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:40:40.489448 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:40:40.489417 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7d56db_479e_40fe_9d4f_dc0842e7a4b7.slice/crio-f800f156a9b259b304105b503a9c293aaef46dab426f5cb53774719cb224b724 WatchSource:0}: Error finding container f800f156a9b259b304105b503a9c293aaef46dab426f5cb53774719cb224b724: Status 404 returned error can't find the container with id f800f156a9b259b304105b503a9c293aaef46dab426f5cb53774719cb224b724 Sep 16 19:40:41.204794 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:41.204746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerStarted","Data":"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a"} Sep 16 19:40:41.205369 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:41.204819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerStarted","Data":"903a63f2a57baadde5ca097b5511973fbd9debf14a9145927d3ca3d87a3bfb1a"} Sep 16 19:40:41.207612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:41.207577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerStarted","Data":"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005"} Sep 16 19:40:41.207733 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:41.207636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerStarted","Data":"f800f156a9b259b304105b503a9c293aaef46dab426f5cb53774719cb224b724"} Sep 16 19:40:43.222994 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:43.222958 2565 generic.go:356] "Generic (PLEG): container finished" podID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerID="f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005" exitCode=0 Sep 16 19:40:43.223639 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:43.223042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerDied","Data":"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005"} Sep 16 19:40:43.224955 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:43.224921 2565 generic.go:356] "Generic (PLEG): container finished" podID="506f23ea-ac27-4142-a2db-ea49b874d727" containerID="7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a" exitCode=0 Sep 16 19:40:43.225046 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:43.224960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerDied","Data":"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a"} Sep 16 19:40:44.234021 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.233970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerStarted","Data":"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e"} Sep 16 19:40:44.236072 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.236042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerStarted","Data":"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857"} Sep 16 19:40:44.236273 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.236254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:40:44.260910 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.260851 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" podStartSLOduration=2.606169789 podStartE2EDuration="5.260830895s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.654661106s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:44.257687658 +0000 UTC m=+2128.463643413" watchObservedRunningTime="2026-09-16 19:40:44.260830895 +0000 UTC m=+2128.466786648" Sep 16 19:40:44.279099 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.279047 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" podStartSLOduration=2.684658901 podStartE2EDuration="5.279032618s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.594373717s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:44.278483546 +0000 UTC m=+2128.484439324" watchObservedRunningTime="2026-09-16 19:40:44.279032618 +0000 UTC m=+2128.484988370" Sep 16 19:40:44.906572 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:44.906539 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:49.907455 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:49.907417 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:49.922736 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:49.922709 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:40:50.286318 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:40:50.286267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:41:15.254729 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:15.254692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:41:16.402752 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:16.402723 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s_4f7d56db-479e-40fe-9d4f-dc0842e7a4b7/main/0.log" Sep 16 19:41:18.622233 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:18.622196 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:41:18.626319 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:18.626293 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:41:18.626566 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:18.626525 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="main" containerID="cri-o://d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857" gracePeriod=30 Sep 16 19:41:18.632420 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:18.632380 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:41:18.632420 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:41:18.632420 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:41:18.632420 ip-10-0-128-148 kubenswrapper[2565]: > containerID="0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" cmd=["/bin/sleep","15"] Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:18.632446 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" message="" err=< Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:18.632462 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="main" containerID="cri-o://0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" Sep 16 19:41:18.632664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:18.632537 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="main" containerID="cri-o://0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" gracePeriod=60 Sep 16 19:41:19.002037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.002010 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:41:19.047458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047423 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047467 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047458 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047495 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29d66\" (UniqueName: \"kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047526 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047567 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047657 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047727 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location\") pod \"506f23ea-ac27-4142-a2db-ea49b874d727\" (UID: \"506f23ea-ac27-4142-a2db-ea49b874d727\") " Sep 16 19:41:19.047802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home" (OuterVolumeSpecName: "home") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.048107 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.047970 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.048207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.048147 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.048207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.048188 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache" (OuterVolumeSpecName: "model-cache") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.048522 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.048500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:41:19.050656 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.050628 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm" (OuterVolumeSpecName: "dshm") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.050777 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.050650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66" (OuterVolumeSpecName: "kube-api-access-29d66") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "kube-api-access-29d66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:41:19.050777 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.050744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:41:19.072791 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.072755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "506f23ea-ac27-4142-a2db-ea49b874d727" (UID: "506f23ea-ac27-4142-a2db-ea49b874d727"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.148484 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148452 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148484 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148481 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148493 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/506f23ea-ac27-4142-a2db-ea49b874d727-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148504 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148512 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506f23ea-ac27-4142-a2db-ea49b874d727-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148520 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29d66\" (UniqueName: \"kubernetes.io/projected/506f23ea-ac27-4142-a2db-ea49b874d727-kube-api-access-29d66\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.148640 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.148529 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506f23ea-ac27-4142-a2db-ea49b874d727-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.271349 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.271320 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:41:19.349681 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.349652 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert\") pod \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " Sep 16 19:41:19.349858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.349723 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs\") pod \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " Sep 16 19:41:19.349858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.349754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location\") pod \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " Sep 16 19:41:19.349858 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.349827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir\") pod \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\" (UID: \"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7\") " Sep 16 19:41:19.350101 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.350077 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" (UID: "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:41:19.350234 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.350211 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" (UID: "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.351965 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.351944 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" (UID: "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:41:19.371944 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.371910 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" (UID: "4f7d56db-479e-40fe-9d4f-dc0842e7a4b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:41:19.450903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.450809 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.450903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.450842 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.450903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.450854 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.450903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.450868 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:41:19.451332 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.451256 2565 generic.go:356] "Generic (PLEG): container finished" podID="506f23ea-ac27-4142-a2db-ea49b874d727" containerID="0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" exitCode=0 Sep 16 19:41:19.451388 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.451323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerDied","Data":"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e"} Sep 16 19:41:19.451388 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.451342 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" Sep 16 19:41:19.451482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.451392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv" event={"ID":"506f23ea-ac27-4142-a2db-ea49b874d727","Type":"ContainerDied","Data":"903a63f2a57baadde5ca097b5511973fbd9debf14a9145927d3ca3d87a3bfb1a"} Sep 16 19:41:19.451482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.451425 2565 scope.go:117] "RemoveContainer" containerID="0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" Sep 16 19:41:19.453310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.453241 2565 generic.go:356] "Generic (PLEG): container finished" podID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerID="d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857" exitCode=0 Sep 16 19:41:19.453310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.453293 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerDied","Data":"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857"} Sep 16 19:41:19.453310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.453304 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" Sep 16 19:41:19.453310 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.453336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s" event={"ID":"4f7d56db-479e-40fe-9d4f-dc0842e7a4b7","Type":"ContainerDied","Data":"f800f156a9b259b304105b503a9c293aaef46dab426f5cb53774719cb224b724"} Sep 16 19:41:19.463166 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.463148 2565 scope.go:117] "RemoveContainer" containerID="7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a" Sep 16 19:41:19.471800 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.471780 2565 scope.go:117] "RemoveContainer" containerID="0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" Sep 16 19:41:19.472086 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:19.472065 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e\": container with ID starting with 0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e not found: ID does not exist" containerID="0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e" Sep 16 19:41:19.472150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.472094 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e"} err="failed to get container status \"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e\": rpc error: code = NotFound desc = could not find container \"0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e\": container with ID starting with 0acc4756a817403463a564b14f5b63417b14b76b16fb13138055167f8d4ee84e not found: ID does not exist" Sep 16 19:41:19.472150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.472109 2565 scope.go:117] "RemoveContainer" containerID="7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a" Sep 16 19:41:19.472353 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:19.472339 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a\": container with ID starting with 7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a not found: ID does not exist" containerID="7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a" Sep 16 19:41:19.472404 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.472357 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a"} err="failed to get container status \"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a\": rpc error: code = NotFound desc = could not find container \"7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a\": container with ID starting with 7abfb3b2f7756e1a4c6b3962e4e2abbb56eb34a2e658dc41e0b855d71833949a not found: ID does not exist" Sep 16 19:41:19.472404 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.472368 2565 scope.go:117] "RemoveContainer" containerID="d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857" Sep 16 19:41:19.480816 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.480795 2565 scope.go:117] "RemoveContainer" containerID="f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005" Sep 16 19:41:19.487520 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.487495 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:41:19.491114 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.491095 2565 scope.go:117] "RemoveContainer" containerID="d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857" Sep 16 19:41:19.491458 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:19.491440 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857\": container with ID starting with d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857 not found: ID does not exist" containerID="d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857" Sep 16 19:41:19.491514 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.491467 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857"} err="failed to get container status \"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857\": rpc error: code = NotFound desc = could not find container \"d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857\": container with ID starting with d16be63eb2908f564d4fc021296a0c1eba5f3027cd73adb0a09addfd9d288857 not found: ID does not exist" Sep 16 19:41:19.491514 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.491485 2565 scope.go:117] "RemoveContainer" containerID="f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005" Sep 16 19:41:19.491784 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:41:19.491766 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005\": container with ID starting with f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005 not found: ID does not exist" containerID="f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005" Sep 16 19:41:19.491847 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.491790 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005"} err="failed to get container status \"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005\": rpc error: code = NotFound desc = could not find container \"f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005\": container with ID starting with f5c57bc3c0864074c3c2def8b305353e0a9bd4ca3441c80d9070d2bdbd706005 not found: ID does not exist" Sep 16 19:41:19.493457 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.493438 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-85b6d5d689-xt74s"] Sep 16 19:41:19.510329 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.510300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:41:19.512439 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:19.512422 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7fd495d9b7-qz5qv"] Sep 16 19:41:20.254198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:20.254161 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" path="/var/lib/kubelet/pods/4f7d56db-479e-40fe-9d4f-dc0842e7a4b7/volumes" Sep 16 19:41:20.254706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:20.254690 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" path="/var/lib/kubelet/pods/506f23ea-ac27-4142-a2db-ea49b874d727/volumes" Sep 16 19:41:51.298014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.297982 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298430 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="storage-initializer" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298443 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="storage-initializer" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298460 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="main" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298465 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="main" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298478 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="main" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298484 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="main" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298494 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="storage-initializer" Sep 16 19:41:51.298513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298499 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="storage-initializer" Sep 16 19:41:51.298770 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298552 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f7d56db-479e-40fe-9d4f-dc0842e7a4b7" containerName="main" Sep 16 19:41:51.298770 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.298561 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="506f23ea-ac27-4142-a2db-ea49b874d727" containerName="main" Sep 16 19:41:51.301485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.301468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.305971 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.305882 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\"" Sep 16 19:41:51.305971 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.305952 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\"" Sep 16 19:41:51.306903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.306883 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-tmsl4\"" Sep 16 19:41:51.307080 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.307014 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\"" Sep 16 19:41:51.323891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.323863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:41:51.430903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.430864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.430903 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.430899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.431143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.430925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trjv\" (UniqueName: \"kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.431143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.430959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.431143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.431042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.431143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.431065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531899 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531899 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531899 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trjv\" (UniqueName: \"kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.531899 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.531840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.532165 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.532080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.532216 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.532169 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.532252 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.532232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.534251 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.534232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.534446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.534427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.540998 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.540971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trjv\" (UniqueName: \"kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv\") pod \"leave-v1-kserve-65684d4f5-tc6qz\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.613117 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.613026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:41:51.900126 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:51.900103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:41:51.901843 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:41:51.901813 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2121f1c_6e5a_451d_86c7_24f55747f32f.slice/crio-309ebdf90d0f477353797ce47a71a1f175e68c802003a23d7cc23673b5a73baf WatchSource:0}: Error finding container 309ebdf90d0f477353797ce47a71a1f175e68c802003a23d7cc23673b5a73baf: Status 404 returned error can't find the container with id 309ebdf90d0f477353797ce47a71a1f175e68c802003a23d7cc23673b5a73baf Sep 16 19:41:52.644901 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:52.644864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" event={"ID":"e2121f1c-6e5a-451d-86c7-24f55747f32f","Type":"ContainerStarted","Data":"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e"} Sep 16 19:41:52.645316 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:52.644921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" event={"ID":"e2121f1c-6e5a-451d-86c7-24f55747f32f","Type":"ContainerStarted","Data":"309ebdf90d0f477353797ce47a71a1f175e68c802003a23d7cc23673b5a73baf"} Sep 16 19:41:52.681047 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:52.680992 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" podStartSLOduration=1.680975154 podStartE2EDuration="1.680975154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:52.680265864 +0000 UTC m=+2196.886221617" watchObservedRunningTime="2026-09-16 19:41:52.680975154 +0000 UTC m=+2196.886931100" Sep 16 19:41:53.614070 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:41:53.614018 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:42:01.613550 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:01.613515 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:42:01.626379 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:01.626348 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:42:01.707254 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:01.707222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:42:53.482823 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.482782 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:42:53.486416 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.486398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.489712 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.489691 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\"" Sep 16 19:42:53.489836 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.489691 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-czpmt\"" Sep 16 19:42:53.490048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.490032 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\"" Sep 16 19:42:53.490523 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.490505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\"" Sep 16 19:42:53.490599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.490566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:42:53.490722 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.490706 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\"" Sep 16 19:42:53.505738 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.505710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:42:53.552142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552325 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.552446 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.552405 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqq2b\" (UniqueName: \"kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.653672 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.653842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.653842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.653842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqq2b\" (UniqueName: \"kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.653980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654109 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654409 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654409 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654362 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654599 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654761 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654739 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.654828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.654793 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.656350 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.656331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.656437 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.656421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.663044 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.663025 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqq2b\" (UniqueName: \"kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b\") pod \"tokenizer-template-args-test-kserve-84f74d478f-v9nmt\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.716252 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.716220 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:42:53.721305 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.721269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.734385 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.734323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:42:53.755489 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.755456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.755671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.755546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.755671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.755601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.755671 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.755642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.797218 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.797179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:53.857274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.857224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.857610 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.857578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.857811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.857792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.857990 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.857968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.858162 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.858003 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.858315 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.857661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.859094 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.859066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:53.860904 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:53.860863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:54.035189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:54.035102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:54.079194 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:54.079169 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:42:54.082522 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:42:54.082486 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23eb47db_9336_45d2_8a58_78ea69de2c69.slice/crio-ed840c1cd9f92010bbff6b50e7143ae64bc00de57f3e693f083f0a399954b4ef WatchSource:0}: Error finding container ed840c1cd9f92010bbff6b50e7143ae64bc00de57f3e693f083f0a399954b4ef: Status 404 returned error can't find the container with id ed840c1cd9f92010bbff6b50e7143ae64bc00de57f3e693f083f0a399954b4ef Sep 16 19:42:54.336595 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:54.336565 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:42:54.337942 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:42:54.337915 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1e0a8b_3eb3_46fe_b2a2_1db26116ceb5.slice/crio-fb7d1aacf5332b2cc5048b90a9bee4440cb118618f3a223c2e8aff806fb394ea WatchSource:0}: Error finding container fb7d1aacf5332b2cc5048b90a9bee4440cb118618f3a223c2e8aff806fb394ea: Status 404 returned error can't find the container with id fb7d1aacf5332b2cc5048b90a9bee4440cb118618f3a223c2e8aff806fb394ea Sep 16 19:42:55.016721 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:55.016681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerStarted","Data":"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a"} Sep 16 19:42:55.017186 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:55.016752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerStarted","Data":"fb7d1aacf5332b2cc5048b90a9bee4440cb118618f3a223c2e8aff806fb394ea"} Sep 16 19:42:55.018891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:55.018863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerStarted","Data":"8a6717e023565e94a884f1c3ae459fad100bd933fd0b63a475e3be1585941974"} Sep 16 19:42:55.019016 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:55.018925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerStarted","Data":"ed840c1cd9f92010bbff6b50e7143ae64bc00de57f3e693f083f0a399954b4ef"} Sep 16 19:42:57.035873 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:57.035837 2565 generic.go:356] "Generic (PLEG): container finished" podID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerID="8a6717e023565e94a884f1c3ae459fad100bd933fd0b63a475e3be1585941974" exitCode=0 Sep 16 19:42:57.036267 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:57.035910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerDied","Data":"8a6717e023565e94a884f1c3ae459fad100bd933fd0b63a475e3be1585941974"} Sep 16 19:42:58.043717 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:58.043678 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerID="b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a" exitCode=0 Sep 16 19:42:58.044140 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:58.043750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerDied","Data":"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a"} Sep 16 19:42:58.045779 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:58.045756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerStarted","Data":"afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e"} Sep 16 19:42:58.122472 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:58.122427 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" podStartSLOduration=2.494535292 podStartE2EDuration="5.122410631s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.627875339s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:58.115629921 +0000 UTC m=+2262.321585674" watchObservedRunningTime="2026-09-16 19:42:58.122410631 +0000 UTC m=+2262.328366384" Sep 16 19:42:58.797680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:58.797635 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:42:59.053687 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:59.053604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerStarted","Data":"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02"} Sep 16 19:42:59.054118 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:59.053916 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:42:59.079307 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:42:59.079220 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" podStartSLOduration=3.461630979 podStartE2EDuration="6.079199238s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.617568259s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:59.075009216 +0000 UTC m=+2263.280964969" watchObservedRunningTime="2026-09-16 19:42:59.079199238 +0000 UTC m=+2263.285154993" Sep 16 19:43:03.797496 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:03.797457 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:43:03.810744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:03.810717 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:43:04.095373 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:04.095293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:43:11.758757 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:11.758724 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:43:26.786033 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:26.785987 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" podUID="e2121f1c-6e5a-451d-86c7-24f55747f32f" containerName="main" containerID="cri-o://9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e" gracePeriod=45 Sep 16 19:43:27.063657 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.063631 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:43:27.083934 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.083897 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.083934 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.083933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.084189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.083956 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.084393 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.084349 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:43:27.084592 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.084558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.084973 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.084736 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:43:27.085108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.085091 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.085228 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.085213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2trjv\" (UniqueName: \"kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv\") pod \"e2121f1c-6e5a-451d-86c7-24f55747f32f\" (UID: \"e2121f1c-6e5a-451d-86c7-24f55747f32f\") " Sep 16 19:43:27.086133 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.086103 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.086261 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.086249 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.086360 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.085761 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home" (OuterVolumeSpecName: "home") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:43:27.087763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.087727 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:43:27.087763 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.087735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm" (OuterVolumeSpecName: "dshm") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:43:27.088772 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.088739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv" (OuterVolumeSpecName: "kube-api-access-2trjv") pod "e2121f1c-6e5a-451d-86c7-24f55747f32f" (UID: "e2121f1c-6e5a-451d-86c7-24f55747f32f"). InnerVolumeSpecName "kube-api-access-2trjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:43:27.187582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.187543 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.187582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.187578 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trjv\" (UniqueName: \"kubernetes.io/projected/e2121f1c-6e5a-451d-86c7-24f55747f32f-kube-api-access-2trjv\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.187582 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.187591 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2121f1c-6e5a-451d-86c7-24f55747f32f-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.187854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.187602 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2121f1c-6e5a-451d-86c7-24f55747f32f-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:43:27.242175 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.242132 2565 generic.go:356] "Generic (PLEG): container finished" podID="e2121f1c-6e5a-451d-86c7-24f55747f32f" containerID="9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e" exitCode=0 Sep 16 19:43:27.242389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.242191 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" Sep 16 19:43:27.242389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.242227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" event={"ID":"e2121f1c-6e5a-451d-86c7-24f55747f32f","Type":"ContainerDied","Data":"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e"} Sep 16 19:43:27.242389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.242303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz" event={"ID":"e2121f1c-6e5a-451d-86c7-24f55747f32f","Type":"ContainerDied","Data":"309ebdf90d0f477353797ce47a71a1f175e68c802003a23d7cc23673b5a73baf"} Sep 16 19:43:27.242389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.242332 2565 scope.go:117] "RemoveContainer" containerID="9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e" Sep 16 19:43:27.253103 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.253077 2565 scope.go:117] "RemoveContainer" containerID="9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e" Sep 16 19:43:27.253456 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:43:27.253434 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e\": container with ID starting with 9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e not found: ID does not exist" containerID="9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e" Sep 16 19:43:27.253546 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.253464 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e"} err="failed to get container status \"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e\": rpc error: code = NotFound desc = could not find container \"9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e\": container with ID starting with 9b426e72477484c7ada0f54cdeb7771de66ce40a8635f5e4fea3a0848135e62e not found: ID does not exist" Sep 16 19:43:27.267400 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.267365 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:43:27.273552 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:27.273524 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tc6qz"] Sep 16 19:43:28.255978 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:28.255942 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2121f1c-6e5a-451d-86c7-24f55747f32f" path="/var/lib/kubelet/pods/e2121f1c-6e5a-451d-86c7-24f55747f32f/volumes" Sep 16 19:43:30.061996 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:30.061949 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused" Sep 16 19:43:40.069868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:43:40.069833 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:44:05.185346 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:05.185312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs_4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5/main/0.log" Sep 16 19:44:39.299750 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.299670 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:44:39.300277 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.300025 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="main" containerID="cri-o://02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02" gracePeriod=30 Sep 16 19:44:39.304491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.304464 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:44:39.314504 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:44:39.314469 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:44:39.314504 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:44:39.314504 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:44:39.314504 ip-10-0-128-148 kubenswrapper[2565]: > containerID="afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e" cmd=["/bin/sleep","15"] Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.314529 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" message="" err=< Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:44:39.314547 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="main" containerID="cri-o://afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e" Sep 16 19:44:39.314886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.314588 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="main" containerID="cri-o://afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e" gracePeriod=60 Sep 16 19:44:39.665751 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.665714 2565 generic.go:356] "Generic (PLEG): container finished" podID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerID="afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e" exitCode=0 Sep 16 19:44:39.665922 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.665793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerDied","Data":"afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e"} Sep 16 19:44:39.683766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.683739 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:44:39.765225 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765225 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765242 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765361 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqq2b\" (UniqueName: \"kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765411 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765528 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.765817 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.765535 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert\") pod \"23eb47db-9336-45d2-8a58-78ea69de2c69\" (UID: \"23eb47db-9336-45d2-8a58-78ea69de2c69\") " Sep 16 19:44:39.766093 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.766059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home" (OuterVolumeSpecName: "home") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.766359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.766334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache" (OuterVolumeSpecName: "model-cache") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.766504 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.766479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.766568 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.766551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:44:39.768933 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.768899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm" (OuterVolumeSpecName: "dshm") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.769238 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.769208 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b" (OuterVolumeSpecName: "kube-api-access-sqq2b") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "kube-api-access-sqq2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:44:39.769380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.769310 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:44:39.788075 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.788039 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23eb47db-9336-45d2-8a58-78ea69de2c69" (UID: "23eb47db-9336-45d2-8a58-78ea69de2c69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.866334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866302 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866334 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866335 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866346 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866357 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqq2b\" (UniqueName: \"kubernetes.io/projected/23eb47db-9336-45d2-8a58-78ea69de2c69-kube-api-access-sqq2b\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866364 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866372 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb47db-9336-45d2-8a58-78ea69de2c69-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866398 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb47db-9336-45d2-8a58-78ea69de2c69-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.866490 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.866406 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23eb47db-9336-45d2-8a58-78ea69de2c69-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.870611 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.870585 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:44:39.967694 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.967664 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert\") pod \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " Sep 16 19:44:39.967865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.967783 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir\") pod \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " Sep 16 19:44:39.967865 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.967812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs\") pod \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " Sep 16 19:44:39.967979 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.967880 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location\") pod \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\" (UID: \"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5\") " Sep 16 19:44:39.968062 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.968041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" (UID: "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:44:39.968189 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.968174 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:39.968301 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.968261 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" (UID: "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:39.970069 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.970047 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" (UID: "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:44:39.990848 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:39.990810 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" (UID: "4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:44:40.068841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.068808 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:40.068841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.068835 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:40.068841 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.068845 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:44:40.673098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.673006 2565 generic.go:356] "Generic (PLEG): container finished" podID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerID="02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02" exitCode=0 Sep 16 19:44:40.673098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.673063 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" Sep 16 19:44:40.673098 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.673089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerDied","Data":"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02"} Sep 16 19:44:40.673689 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.673146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs" event={"ID":"4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5","Type":"ContainerDied","Data":"fb7d1aacf5332b2cc5048b90a9bee4440cb118618f3a223c2e8aff806fb394ea"} Sep 16 19:44:40.673689 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.673179 2565 scope.go:117] "RemoveContainer" containerID="02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02" Sep 16 19:44:40.675397 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.675357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" event={"ID":"23eb47db-9336-45d2-8a58-78ea69de2c69","Type":"ContainerDied","Data":"ed840c1cd9f92010bbff6b50e7143ae64bc00de57f3e693f083f0a399954b4ef"} Sep 16 19:44:40.675609 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.675406 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt" Sep 16 19:44:40.683605 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.683572 2565 scope.go:117] "RemoveContainer" containerID="b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a" Sep 16 19:44:40.693623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.693588 2565 scope.go:117] "RemoveContainer" containerID="02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:44:40.694149 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02\": container with ID starting with 02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02 not found: ID does not exist" containerID="02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.694227 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02"} err="failed to get container status \"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02\": rpc error: code = NotFound desc = could not find container \"02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02\": container with ID starting with 02eccdda6648ca6a3d6cf9c5e101ce426a8bd09b5a79d6c183e8f7fd69df0d02 not found: ID does not exist" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.694260 2565 scope.go:117] "RemoveContainer" containerID="b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:44:40.694618 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a\": container with ID starting with b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a not found: ID does not exist" containerID="b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.694655 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a"} err="failed to get container status \"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a\": rpc error: code = NotFound desc = could not find container \"b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a\": container with ID starting with b45d9cc248b72e7921ce5b301e7af67567f94c59fd2e07924a46be1918d66d3a not found: ID does not exist" Sep 16 19:44:40.694764 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.694677 2565 scope.go:117] "RemoveContainer" containerID="afb511a40fc24784733b969aa2228016642aa9bbc11d1d42970fa358b96d174e" Sep 16 19:44:40.697221 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.697197 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:44:40.706510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.706488 2565 scope.go:117] "RemoveContainer" containerID="8a6717e023565e94a884f1c3ae459fad100bd933fd0b63a475e3be1585941974" Sep 16 19:44:40.707859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.707830 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6987f6fdb5-8rpzs"] Sep 16 19:44:40.726292 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.726247 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:44:40.731485 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:40.731465 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84f74d478f-v9nmt"] Sep 16 19:44:42.253486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:42.253447 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" path="/var/lib/kubelet/pods/23eb47db-9336-45d2-8a58-78ea69de2c69/volumes" Sep 16 19:44:42.253899 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:44:42.253885 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" path="/var/lib/kubelet/pods/4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5/volumes" Sep 16 19:45:43.082177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082088 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082508 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082520 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082541 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="storage-initializer" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082547 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="storage-initializer" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082554 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2121f1c-6e5a-451d-86c7-24f55747f32f" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082560 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2121f1c-6e5a-451d-86c7-24f55747f32f" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082566 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="storage-initializer" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082571 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="storage-initializer" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082592 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="main" Sep 16 19:45:43.082644 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082648 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="23eb47db-9336-45d2-8a58-78ea69de2c69" containerName="main" Sep 16 19:45:43.083034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082656 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1e0a8b-3eb3-46fe-b2a2-1db26116ceb5" containerName="main" Sep 16 19:45:43.083034 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.082662 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2121f1c-6e5a-451d-86c7-24f55747f32f" containerName="main" Sep 16 19:45:43.085738 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.085716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.088415 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.088393 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\"" Sep 16 19:45:43.089338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.089317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\"" Sep 16 19:45:43.089436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.089335 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9p7tz\"" Sep 16 19:45:43.089436 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.089375 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\"" Sep 16 19:45:43.095129 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.095106 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:45:43.147682 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.147844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6sqb\" (UniqueName: \"kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.147844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.147844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.147964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.147964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.147887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6sqb\" (UniqueName: \"kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.248859 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.248825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.249142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.249121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.249214 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.249192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.249274 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.249200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.251199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.251176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.251538 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.251523 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.257618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.257592 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6sqb\" (UniqueName: \"kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.397013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.396907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:43.676316 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:43.676267 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:45:43.677795 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:45:43.677767 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ffaf84_4637_4d5d_9144_eb3a77967175.slice/crio-d7402451305ab130dd07cd44db8c9388605e718e6fb22944f4fc7937e47472dc WatchSource:0}: Error finding container d7402451305ab130dd07cd44db8c9388605e718e6fb22944f4fc7937e47472dc: Status 404 returned error can't find the container with id d7402451305ab130dd07cd44db8c9388605e718e6fb22944f4fc7937e47472dc Sep 16 19:45:44.038398 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:44.038355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" event={"ID":"82ffaf84-4637-4d5d-9144-eb3a77967175","Type":"ContainerStarted","Data":"d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307"} Sep 16 19:45:44.038588 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:44.038416 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" event={"ID":"82ffaf84-4637-4d5d-9144-eb3a77967175","Type":"ContainerStarted","Data":"d7402451305ab130dd07cd44db8c9388605e718e6fb22944f4fc7937e47472dc"} Sep 16 19:45:44.058227 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:44.058181 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" podStartSLOduration=1.058166132 podStartE2EDuration="1.058166132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:44.056263277 +0000 UTC m=+2428.262219023" watchObservedRunningTime="2026-09-16 19:45:44.058166132 +0000 UTC m=+2428.264121884" Sep 16 19:45:44.397739 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:44.397655 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:53.397516 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:53.397474 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:53.410443 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:53.410418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:45:54.107182 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:45:54.107145 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:46:17.960902 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:17.960864 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:46:17.972195 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:46:17.972155 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:46:17.972195 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:46:17.972195 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:46:17.972195 ip-10-0-128-148 kubenswrapper[2565]: > containerID="d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307" cmd=["/bin/sleep","15"] Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:17.972219 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" message="" err=< Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:46:17.972234 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" containerID="cri-o://d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307" Sep 16 19:46:17.972647 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:17.972308 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" containerID="cri-o://d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307" gracePeriod=60 Sep 16 19:46:18.096581 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.096547 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused" Sep 16 19:46:18.233364 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.233327 2565 generic.go:356] "Generic (PLEG): container finished" podID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerID="d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307" exitCode=0 Sep 16 19:46:18.233532 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.233388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" event={"ID":"82ffaf84-4637-4d5d-9144-eb3a77967175","Type":"ContainerDied","Data":"d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307"} Sep 16 19:46:18.333238 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.333211 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:46:18.476677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476642 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.476677 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.476936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476701 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.476936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476733 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.476936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476762 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6sqb\" (UniqueName: \"kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.476936 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.476801 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs\") pod \"82ffaf84-4637-4d5d-9144-eb3a77967175\" (UID: \"82ffaf84-4637-4d5d-9144-eb3a77967175\") " Sep 16 19:46:18.477141 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477022 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:46:18.477141 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477038 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache" (OuterVolumeSpecName: "model-cache") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:46:18.477141 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home" (OuterVolumeSpecName: "home") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:46:18.477265 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477174 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:18.477265 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477204 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:18.477265 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.477214 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:18.479206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.479179 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb" (OuterVolumeSpecName: "kube-api-access-f6sqb") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "kube-api-access-f6sqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:46:18.479318 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.479232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm" (OuterVolumeSpecName: "dshm") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:46:18.479318 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.479244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82ffaf84-4637-4d5d-9144-eb3a77967175" (UID: "82ffaf84-4637-4d5d-9144-eb3a77967175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:46:18.577811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.577773 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ffaf84-4637-4d5d-9144-eb3a77967175-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:18.577811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.577808 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6sqb\" (UniqueName: \"kubernetes.io/projected/82ffaf84-4637-4d5d-9144-eb3a77967175-kube-api-access-f6sqb\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:18.577811 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:18.577819 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ffaf84-4637-4d5d-9144-eb3a77967175-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:46:19.242881 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:19.242614 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" event={"ID":"82ffaf84-4637-4d5d-9144-eb3a77967175","Type":"ContainerDied","Data":"d7402451305ab130dd07cd44db8c9388605e718e6fb22944f4fc7937e47472dc"} Sep 16 19:46:19.242881 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:19.242686 2565 scope.go:117] "RemoveContainer" containerID="d9c93a0678813a78251ceb10d35b163fb664f1ce85d533a8eb9b20e010d1f307" Sep 16 19:46:19.242881 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:19.242845 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4" Sep 16 19:46:19.282813 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:19.282781 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:46:19.287533 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:19.287505 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-5cwm4"] Sep 16 19:46:20.253607 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:20.253572 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" path="/var/lib/kubelet/pods/82ffaf84-4637-4d5d-9144-eb3a77967175/volumes" Sep 16 19:46:39.396799 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.396756 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:46:39.397396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.397379 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" Sep 16 19:46:39.397396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.397399 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" Sep 16 19:46:39.397591 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.397489 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ffaf84-4637-4d5d-9144-eb3a77967175" containerName="main" Sep 16 19:46:39.401678 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.401661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.408498 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.408180 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\"" Sep 16 19:46:39.409118 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.409097 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\"" Sep 16 19:46:39.409524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.409490 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\"" Sep 16 19:46:39.409524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.409515 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-2d9s2\"" Sep 16 19:46:39.415888 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.415862 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:46:39.424450 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.424959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8sn\" (UniqueName: \"kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.424959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.424959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.424959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.424959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.424666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525727 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8sn\" (UniqueName: \"kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.525908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.525873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.526224 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.526158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.526309 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.526222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.526359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.526310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.528368 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.528343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.528538 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.528522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.537167 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.537140 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8sn\" (UniqueName: \"kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn\") pod \"late-v2-kserve-86766546f-p82kh\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.712592 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.712500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:39.994268 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:39.994238 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:46:39.996409 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:46:39.996381 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf021726b_452f_46f4_be26_794d73c56c0a.slice/crio-68870796c9a5cc704083e6dba55677bbbd8de0a66ac16f51fe832ddece9cbb47 WatchSource:0}: Error finding container 68870796c9a5cc704083e6dba55677bbbd8de0a66ac16f51fe832ddece9cbb47: Status 404 returned error can't find the container with id 68870796c9a5cc704083e6dba55677bbbd8de0a66ac16f51fe832ddece9cbb47 Sep 16 19:46:40.364516 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:40.364428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" event={"ID":"f021726b-452f-46f4-be26-794d73c56c0a","Type":"ContainerStarted","Data":"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d"} Sep 16 19:46:40.364516 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:40.364495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" event={"ID":"f021726b-452f-46f4-be26-794d73c56c0a","Type":"ContainerStarted","Data":"68870796c9a5cc704083e6dba55677bbbd8de0a66ac16f51fe832ddece9cbb47"} Sep 16 19:46:40.384656 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:40.384603 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" podStartSLOduration=1.38458669 podStartE2EDuration="1.38458669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:40.38252548 +0000 UTC m=+2484.588481248" watchObservedRunningTime="2026-09-16 19:46:40.38458669 +0000 UTC m=+2484.590542442" Sep 16 19:46:40.712920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:40.712810 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:49.712857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:49.712821 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:49.725769 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:49.725745 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:46:50.431915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:46:50.431884 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:47:24.665673 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.665593 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:47:24.669545 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.669525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.673009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.672978 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\"" Sep 16 19:47:24.673145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.673015 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\"" Sep 16 19:47:24.673145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.673022 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\"" Sep 16 19:47:24.673145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.673033 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-d9pzz\"" Sep 16 19:47:24.686245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.686211 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:47:24.827466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.827466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.827684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.827684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4j5\" (UniqueName: \"kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.827684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.827789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.827685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4j5\" (UniqueName: \"kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928683 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.928966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.928737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.929117 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.929073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.929199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.929176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.929263 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.929229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.931357 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.931330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.931482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.931428 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.937698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.937672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4j5\" (UniqueName: \"kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:24.981767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:24.981735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:25.261222 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:25.261195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:47:25.262370 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:47:25.262344 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ddc190_f678_497a_a812_dc5b696c68b2.slice/crio-9c95f6aef98b5b745aa447538857cfb4c0a250bb9ab04c0df4db7153b86f1204 WatchSource:0}: Error finding container 9c95f6aef98b5b745aa447538857cfb4c0a250bb9ab04c0df4db7153b86f1204: Status 404 returned error can't find the container with id 9c95f6aef98b5b745aa447538857cfb4c0a250bb9ab04c0df4db7153b86f1204 Sep 16 19:47:25.627590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:25.627495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" event={"ID":"03ddc190-f678-497a-a812-dc5b696c68b2","Type":"ContainerStarted","Data":"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de"} Sep 16 19:47:25.627590 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:25.627547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" event={"ID":"03ddc190-f678-497a-a812-dc5b696c68b2","Type":"ContainerStarted","Data":"9c95f6aef98b5b745aa447538857cfb4c0a250bb9ab04c0df4db7153b86f1204"} Sep 16 19:47:25.657438 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:25.657379 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" podStartSLOduration=1.6573653799999999 podStartE2EDuration="1.65736538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:25.656579405 +0000 UTC m=+2529.862535159" watchObservedRunningTime="2026-09-16 19:47:25.65736538 +0000 UTC m=+2529.863321133" Sep 16 19:47:25.981895 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:25.981848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:32.889735 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:32.889696 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:47:34.982706 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:34.982671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:34.995149 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:34.995120 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:35.694621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:35.694592 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:47:47.909921 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:47.909881 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" podUID="f021726b-452f-46f4-be26-794d73c56c0a" containerName="main" containerID="cri-o://72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d" gracePeriod=45 Sep 16 19:47:48.165058 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.165003 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:47:48.247355 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247436 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247557 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m8sn\" (UniqueName: \"kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn\") pod \"f021726b-452f-46f4-be26-794d73c56c0a\" (UID: \"f021726b-452f-46f4-be26-794d73c56c0a\") " Sep 16 19:47:48.247828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247618 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:47:48.247828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:47:48.247828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home" (OuterVolumeSpecName: "home") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:47:48.247956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247888 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.247956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247909 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.247956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.247922 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.249925 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.249896 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:47:48.250037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.249945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn" (OuterVolumeSpecName: "kube-api-access-7m8sn") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "kube-api-access-7m8sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:47:48.250037 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.250014 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm" (OuterVolumeSpecName: "dshm") pod "f021726b-452f-46f4-be26-794d73c56c0a" (UID: "f021726b-452f-46f4-be26-794d73c56c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:47:48.349078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.349045 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f021726b-452f-46f4-be26-794d73c56c0a-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.349078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.349071 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f021726b-452f-46f4-be26-794d73c56c0a-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.349078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.349082 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8sn\" (UniqueName: \"kubernetes.io/projected/f021726b-452f-46f4-be26-794d73c56c0a-kube-api-access-7m8sn\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:47:48.759190 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.759155 2565 generic.go:356] "Generic (PLEG): container finished" podID="f021726b-452f-46f4-be26-794d73c56c0a" containerID="72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d" exitCode=0 Sep 16 19:47:48.759412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.759199 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" Sep 16 19:47:48.759412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.759236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" event={"ID":"f021726b-452f-46f4-be26-794d73c56c0a","Type":"ContainerDied","Data":"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d"} Sep 16 19:47:48.759412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.759312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh" event={"ID":"f021726b-452f-46f4-be26-794d73c56c0a","Type":"ContainerDied","Data":"68870796c9a5cc704083e6dba55677bbbd8de0a66ac16f51fe832ddece9cbb47"} Sep 16 19:47:48.759412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.759338 2565 scope.go:117] "RemoveContainer" containerID="72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d" Sep 16 19:47:48.768842 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.768825 2565 scope.go:117] "RemoveContainer" containerID="72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d" Sep 16 19:47:48.769125 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:47:48.769108 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d\": container with ID starting with 72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d not found: ID does not exist" containerID="72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d" Sep 16 19:47:48.769198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.769130 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d"} err="failed to get container status \"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d\": rpc error: code = NotFound desc = could not find container \"72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d\": container with ID starting with 72bb0f3a9b5c479ed9aad6d59463c248b7b557d166d883fbe3fc088d08a5596d not found: ID does not exist" Sep 16 19:47:48.777728 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.777704 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:47:48.783549 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:48.783525 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-p82kh"] Sep 16 19:47:50.255432 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:47:50.255395 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f021726b-452f-46f4-be26-794d73c56c0a" path="/var/lib/kubelet/pods/f021726b-452f-46f4-be26-794d73c56c0a/volumes" Sep 16 19:48:22.152460 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.152425 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:48:22.153107 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.153091 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f021726b-452f-46f4-be26-794d73c56c0a" containerName="main" Sep 16 19:48:22.153159 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.153111 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f021726b-452f-46f4-be26-794d73c56c0a" containerName="main" Sep 16 19:48:22.153229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.153218 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f021726b-452f-46f4-be26-794d73c56c0a" containerName="main" Sep 16 19:48:22.157245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.157222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.162856 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.162829 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\"" Sep 16 19:48:22.162986 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.162839 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-sm9fz\"" Sep 16 19:48:22.177435 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.177407 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:48:22.262199 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.262170 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d2c\" (UniqueName: \"kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.262397 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.262298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.363776 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.363737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.363987 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.363962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77d2c\" (UniqueName: \"kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.366492 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.366464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.378125 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.378099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d2c\" (UniqueName: \"kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.469440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.469416 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.754142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.754106 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:48:22.756114 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:48:22.756083 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc66e49f_f501_4095_bd31_b67abb08477a.slice/crio-78c4bbf5842151830ededcc387ea88d3251122c91c87e975121cfe6c681c7b61 WatchSource:0}: Error finding container 78c4bbf5842151830ededcc387ea88d3251122c91c87e975121cfe6c681c7b61: Status 404 returned error can't find the container with id 78c4bbf5842151830ededcc387ea88d3251122c91c87e975121cfe6c681c7b61 Sep 16 19:48:22.961851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.961812 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" event={"ID":"dc66e49f-f501-4095-bd31-b67abb08477a","Type":"ContainerStarted","Data":"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7"} Sep 16 19:48:22.962057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.961873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" event={"ID":"dc66e49f-f501-4095-bd31-b67abb08477a","Type":"ContainerStarted","Data":"78c4bbf5842151830ededcc387ea88d3251122c91c87e975121cfe6c681c7b61"} Sep 16 19:48:22.962057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.961992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:48:22.981868 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:22.981813 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podStartSLOduration=0.981797646 podStartE2EDuration="981.797646ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:22.979644731 +0000 UTC m=+2587.185600483" watchObservedRunningTime="2026-09-16 19:48:22.981797646 +0000 UTC m=+2587.187753399" Sep 16 19:48:53.970206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:48:53.970177 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:49:33.709466 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:33.709428 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:49:33.970304 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:33.970245 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:49:43.969767 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:43.969725 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:49:48.730053 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:48.730002 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" containerID="cri-o://9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7" gracePeriod=45 Sep 16 19:49:53.969905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:53.969870 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:49:53.970375 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:49:53.969985 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:50:03.970519 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:03.970477 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:50:13.969886 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:13.969802 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:50:18.978142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:18.978102 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:50:19.116424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.116315 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77d2c\" (UniqueName: \"kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c\") pod \"dc66e49f-f501-4095-bd31-b67abb08477a\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " Sep 16 19:50:19.116603 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.116459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs\") pod \"dc66e49f-f501-4095-bd31-b67abb08477a\" (UID: \"dc66e49f-f501-4095-bd31-b67abb08477a\") " Sep 16 19:50:19.118771 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.118739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc66e49f-f501-4095-bd31-b67abb08477a" (UID: "dc66e49f-f501-4095-bd31-b67abb08477a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:50:19.118882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.118763 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c" (OuterVolumeSpecName: "kube-api-access-77d2c") pod "dc66e49f-f501-4095-bd31-b67abb08477a" (UID: "dc66e49f-f501-4095-bd31-b67abb08477a"). InnerVolumeSpecName "kube-api-access-77d2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:50:19.217150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.217102 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc66e49f-f501-4095-bd31-b67abb08477a-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:19.217150 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.217148 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77d2c\" (UniqueName: \"kubernetes.io/projected/dc66e49f-f501-4095-bd31-b67abb08477a-kube-api-access-77d2c\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:19.636618 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.636580 2565 generic.go:356] "Generic (PLEG): container finished" podID="dc66e49f-f501-4095-bd31-b67abb08477a" containerID="9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7" exitCode=0 Sep 16 19:50:19.636828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.636635 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" Sep 16 19:50:19.636828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.636650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" event={"ID":"dc66e49f-f501-4095-bd31-b67abb08477a","Type":"ContainerDied","Data":"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7"} Sep 16 19:50:19.636828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.636704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn" event={"ID":"dc66e49f-f501-4095-bd31-b67abb08477a","Type":"ContainerDied","Data":"78c4bbf5842151830ededcc387ea88d3251122c91c87e975121cfe6c681c7b61"} Sep 16 19:50:19.636828 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.636735 2565 scope.go:117] "RemoveContainer" containerID="9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7" Sep 16 19:50:19.646876 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.646848 2565 scope.go:117] "RemoveContainer" containerID="9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7" Sep 16 19:50:19.647188 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:50:19.647162 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7\": container with ID starting with 9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7 not found: ID does not exist" containerID="9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7" Sep 16 19:50:19.647269 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.647198 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7"} err="failed to get container status \"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7\": rpc error: code = NotFound desc = could not find container \"9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7\": container with ID starting with 9569ec76c5fb21c6336f667dbfbd37ce50e0211fe3cc20191d88f4757e3cacf7 not found: ID does not exist" Sep 16 19:50:19.659580 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.659555 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:50:19.664013 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:19.663987 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schec66zn"] Sep 16 19:50:20.253059 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:20.253027 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" path="/var/lib/kubelet/pods/dc66e49f-f501-4095-bd31-b67abb08477a/volumes" Sep 16 19:50:22.623068 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:22.623032 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:50:22.632673 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:50:22.632633 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:50:22.632673 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:50:22.632673 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:50:22.632673 ip-10-0-128-148 kubenswrapper[2565]: > containerID="ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" cmd=["/bin/sleep","15"] Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:22.632687 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" message="" err=< Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:50:22.632702 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" containerID="cri-o://ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" Sep 16 19:50:22.633048 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:22.632739 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" containerID="cri-o://ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" gracePeriod=60 Sep 16 19:50:22.684105 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:22.684060 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused" Sep 16 19:50:22.983207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:22.983179 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:50:23.050655 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050627 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.050844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050669 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.050844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050692 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.050844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.050844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4j5\" (UniqueName: \"kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.050844 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm\") pod \"03ddc190-f678-497a-a812-dc5b696c68b2\" (UID: \"03ddc190-f678-497a-a812-dc5b696c68b2\") " Sep 16 19:50:23.051121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050967 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:23.051121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.050998 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home" (OuterVolumeSpecName: "home") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:23.051121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.051062 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:23.051121 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.051107 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.051259 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.051130 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.053008 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.052986 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm" (OuterVolumeSpecName: "dshm") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:50:23.053095 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.053075 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5" (OuterVolumeSpecName: "kube-api-access-9h4j5") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "kube-api-access-9h4j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:50:23.053137 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.053080 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03ddc190-f678-497a-a812-dc5b696c68b2" (UID: "03ddc190-f678-497a-a812-dc5b696c68b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:50:23.152262 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.152179 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ddc190-f678-497a-a812-dc5b696c68b2-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.152262 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.152212 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.152262 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.152223 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4j5\" (UniqueName: \"kubernetes.io/projected/03ddc190-f678-497a-a812-dc5b696c68b2-kube-api-access-9h4j5\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.152262 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.152233 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ddc190-f678-497a-a812-dc5b696c68b2-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:50:23.663218 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.663179 2565 generic.go:356] "Generic (PLEG): container finished" podID="03ddc190-f678-497a-a812-dc5b696c68b2" containerID="ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" exitCode=0 Sep 16 19:50:23.663717 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.663229 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" Sep 16 19:50:23.663717 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.663266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" event={"ID":"03ddc190-f678-497a-a812-dc5b696c68b2","Type":"ContainerDied","Data":"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de"} Sep 16 19:50:23.663717 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.663358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r" event={"ID":"03ddc190-f678-497a-a812-dc5b696c68b2","Type":"ContainerDied","Data":"9c95f6aef98b5b745aa447538857cfb4c0a250bb9ab04c0df4db7153b86f1204"} Sep 16 19:50:23.663717 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.663385 2565 scope.go:117] "RemoveContainer" containerID="ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" Sep 16 19:50:23.673513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.673494 2565 scope.go:117] "RemoveContainer" containerID="ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" Sep 16 19:50:23.673793 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:50:23.673775 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de\": container with ID starting with ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de not found: ID does not exist" containerID="ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de" Sep 16 19:50:23.673851 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.673802 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de"} err="failed to get container status \"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de\": rpc error: code = NotFound desc = could not find container \"ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de\": container with ID starting with ae158d3749c9fa9fae307ff06f391b7845963ee66abd76232ac115177be488de not found: ID does not exist" Sep 16 19:50:23.692955 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.692928 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:50:23.703198 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:23.703169 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hxl6r"] Sep 16 19:50:24.253923 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:50:24.253890 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" path="/var/lib/kubelet/pods/03ddc190-f678-497a-a812-dc5b696c68b2/volumes" Sep 16 19:51:02.648560 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.648524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:51:02.649064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.648964 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" Sep 16 19:51:02.649064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.648978 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" Sep 16 19:51:02.649064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.648987 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" Sep 16 19:51:02.649064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.648992 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" Sep 16 19:51:02.649064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.649061 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ddc190-f678-497a-a812-dc5b696c68b2" containerName="main" Sep 16 19:51:02.649245 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.649073 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc66e49f-f501-4095-bd31-b67abb08477a" containerName="main" Sep 16 19:51:02.652112 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.652090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.654686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.654662 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\"" Sep 16 19:51:02.655533 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.655506 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\"" Sep 16 19:51:02.655661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.655542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\"" Sep 16 19:51:02.655661 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.655574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-bgtl9\"" Sep 16 19:51:02.661474 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.661450 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:51:02.700147 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.700395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.700395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62rh\" (UniqueName: \"kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.700395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.700395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.700395 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.700309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801158 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801391 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801391 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62rh\" (UniqueName: \"kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801678 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801799 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.801932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.801802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.803711 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.803688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.804009 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.803994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.809893 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.809868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62rh\" (UniqueName: \"kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh\") pod \"owner-v1-kserve-75579ff5bc-lgd2j\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:02.963514 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:02.963415 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:03.308734 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:03.308706 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:51:03.310524 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:51:03.310492 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f87e8b_7338_4576_ab58_8c01f6428b6a.slice/crio-6b681c597bf12ddfce9fc57e1f686fcb10958a737cebe23f7f895ab3e6cf587c WatchSource:0}: Error finding container 6b681c597bf12ddfce9fc57e1f686fcb10958a737cebe23f7f895ab3e6cf587c: Status 404 returned error can't find the container with id 6b681c597bf12ddfce9fc57e1f686fcb10958a737cebe23f7f895ab3e6cf587c Sep 16 19:51:03.895705 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:03.895654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" event={"ID":"70f87e8b-7338-4576-ab58-8c01f6428b6a","Type":"ContainerStarted","Data":"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5"} Sep 16 19:51:03.896107 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:03.895721 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" event={"ID":"70f87e8b-7338-4576-ab58-8c01f6428b6a","Type":"ContainerStarted","Data":"6b681c597bf12ddfce9fc57e1f686fcb10958a737cebe23f7f895ab3e6cf587c"} Sep 16 19:51:03.915963 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:03.915911 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" podStartSLOduration=1.915896327 podStartE2EDuration="1.915896327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:03.915041018 +0000 UTC m=+2748.120996771" watchObservedRunningTime="2026-09-16 19:51:03.915896327 +0000 UTC m=+2748.121852071" Sep 16 19:51:03.964014 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:03.963980 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:12.964629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:12.964588 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:12.977186 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:12.977159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:13.962660 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:13.962630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:51:40.068091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.067994 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:51:40.071108 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.071086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.074943 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.074885 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\"" Sep 16 19:51:40.074943 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.074921 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-dl6cj\"" Sep 16 19:51:40.075145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.074961 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\"" Sep 16 19:51:40.075145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.074977 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-8qp57\"" Sep 16 19:51:40.075145 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.074894 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\"" Sep 16 19:51:40.081786 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.081757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:51:40.140537 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.140501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.140703 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.140568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvw5\" (UniqueName: \"kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.242064 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.242028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvw5\" (UniqueName: \"kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.242262 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.242091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.244860 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.244830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.250713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.250689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvw5\" (UniqueName: \"kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.382752 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.382665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:40.657408 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:40.657380 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:51:40.658367 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:51:40.658342 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5b51fb_ddf3_436b_a899_a60617ff128e.slice/crio-b611ec3baa263889917cc94fd2f7a083371ced91f4c98c3b464c44135576a88c WatchSource:0}: Error finding container b611ec3baa263889917cc94fd2f7a083371ced91f4c98c3b464c44135576a88c: Status 404 returned error can't find the container with id b611ec3baa263889917cc94fd2f7a083371ced91f4c98c3b464c44135576a88c Sep 16 19:51:41.107378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:41.107341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" event={"ID":"7a5b51fb-ddf3-436b-a899-a60617ff128e","Type":"ContainerStarted","Data":"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6"} Sep 16 19:51:41.107789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:41.107400 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" event={"ID":"7a5b51fb-ddf3-436b-a899-a60617ff128e","Type":"ContainerStarted","Data":"b611ec3baa263889917cc94fd2f7a083371ced91f4c98c3b464c44135576a88c"} Sep 16 19:51:41.107789 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:41.107441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:51:41.137021 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:51:41.136970 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podStartSLOduration=1.136954025 podStartE2EDuration="1.136954025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:41.132732232 +0000 UTC m=+2785.338687998" watchObservedRunningTime="2026-09-16 19:51:41.136954025 +0000 UTC m=+2785.342909790" Sep 16 19:52:03.835666 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:03.835632 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:52:12.116744 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:12.116711 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:52:18.858383 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:18.858299 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerName="main" containerID="cri-o://2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5" gracePeriod=45 Sep 16 19:52:18.952033 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:18.951990 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused" Sep 16 19:52:19.114716 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.114562 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:52:19.285331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285331 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285337 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285606 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285397 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285606 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285606 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285494 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q62rh\" (UniqueName: \"kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285606 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285546 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache\") pod \"70f87e8b-7338-4576-ab58-8c01f6428b6a\" (UID: \"70f87e8b-7338-4576-ab58-8c01f6428b6a\") " Sep 16 19:52:19.285854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285752 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:52:19.285854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home" (OuterVolumeSpecName: "home") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:52:19.285953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285866 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:19.285953 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285890 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:19.286243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.285958 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:52:19.287871 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.287838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm" (OuterVolumeSpecName: "dshm") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:52:19.287991 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.287923 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:52:19.287991 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.287937 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh" (OuterVolumeSpecName: "kube-api-access-q62rh") pod "70f87e8b-7338-4576-ab58-8c01f6428b6a" (UID: "70f87e8b-7338-4576-ab58-8c01f6428b6a"). InnerVolumeSpecName "kube-api-access-q62rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:52:19.323823 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.323778 2565 generic.go:356] "Generic (PLEG): container finished" podID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerID="2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5" exitCode=0 Sep 16 19:52:19.324003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.323830 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" Sep 16 19:52:19.324003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.323871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" event={"ID":"70f87e8b-7338-4576-ab58-8c01f6428b6a","Type":"ContainerDied","Data":"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5"} Sep 16 19:52:19.324003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.323936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j" event={"ID":"70f87e8b-7338-4576-ab58-8c01f6428b6a","Type":"ContainerDied","Data":"6b681c597bf12ddfce9fc57e1f686fcb10958a737cebe23f7f895ab3e6cf587c"} Sep 16 19:52:19.324003 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.323968 2565 scope.go:117] "RemoveContainer" containerID="2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5" Sep 16 19:52:19.334038 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.334019 2565 scope.go:117] "RemoveContainer" containerID="2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5" Sep 16 19:52:19.334332 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:52:19.334311 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5\": container with ID starting with 2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5 not found: ID does not exist" containerID="2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5" Sep 16 19:52:19.334389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.334352 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5"} err="failed to get container status \"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5\": rpc error: code = NotFound desc = could not find container \"2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5\": container with ID starting with 2b00cac35e3386a20c021f785e9b188c23011a31c3cc9d30cd58795f266ba3f5 not found: ID does not exist" Sep 16 19:52:19.347134 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.347111 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:52:19.352800 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.352779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-lgd2j"] Sep 16 19:52:19.386748 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.386678 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:19.386748 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.386702 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f87e8b-7338-4576-ab58-8c01f6428b6a-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:19.386748 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.386711 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62rh\" (UniqueName: \"kubernetes.io/projected/70f87e8b-7338-4576-ab58-8c01f6428b6a-kube-api-access-q62rh\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:19.386748 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:19.386719 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f87e8b-7338-4576-ab58-8c01f6428b6a-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:52:20.254057 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:52:20.254020 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" path="/var/lib/kubelet/pods/70f87e8b-7338-4576-ab58-8c01f6428b6a/volumes" Sep 16 19:53:08.727867 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.727789 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:08.728356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.728221 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerName="main" Sep 16 19:53:08.728356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.728235 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerName="main" Sep 16 19:53:08.728356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.728338 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f87e8b-7338-4576-ab58-8c01f6428b6a" containerName="main" Sep 16 19:53:08.731335 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.731319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.733999 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.733981 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\"" Sep 16 19:53:08.741507 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.741479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:08.823159 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.823159 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.823410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.823410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnr8f\" (UniqueName: \"kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.823410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.823410 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.823351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.924731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.924956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.924956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.924956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnr8f\" (UniqueName: \"kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.924956 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.925229 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.924982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.925337 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.925310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.925418 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.925375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.925469 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.925448 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.927520 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.927497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.927649 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.927632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:08.936213 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:08.936187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnr8f\" (UniqueName: \"kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:09.043091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:09.043001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:09.319700 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:09.319665 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:09.321589 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:53:09.321563 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54691fbb_5642_4635_b208_5a63fc43d68e.slice/crio-9fb9496bb74139d21cfc23e62ed08b4417d76cca2ea8b096b9343719606bcca2 WatchSource:0}: Error finding container 9fb9496bb74139d21cfc23e62ed08b4417d76cca2ea8b096b9343719606bcca2: Status 404 returned error can't find the container with id 9fb9496bb74139d21cfc23e62ed08b4417d76cca2ea8b096b9343719606bcca2 Sep 16 19:53:09.609389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:09.609262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" event={"ID":"54691fbb-5642-4635-b208-5a63fc43d68e","Type":"ContainerStarted","Data":"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d"} Sep 16 19:53:09.609389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:09.609358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" event={"ID":"54691fbb-5642-4635-b208-5a63fc43d68e","Type":"ContainerStarted","Data":"9fb9496bb74139d21cfc23e62ed08b4417d76cca2ea8b096b9343719606bcca2"} Sep 16 19:53:09.631253 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:09.631197 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" podStartSLOduration=1.631180418 podStartE2EDuration="1.631180418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:09.627865135 +0000 UTC m=+2873.833820889" watchObservedRunningTime="2026-09-16 19:53:09.631180418 +0000 UTC m=+2873.837136213" Sep 16 19:53:10.044061 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:10.044019 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:19.043780 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:19.043738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:19.056313 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:19.056291 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:19.675664 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:19.675632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:46.004877 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.004844 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:46.015189 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:53:46.015136 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 19:53:46.015189 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:46.015189 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:53:46.015189 ip-10-0-128-148 kubenswrapper[2565]: > containerID="5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" cmd=["/bin/sleep","15"] Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.015190 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" message="" err=< Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:53:46.015206 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" containerName="main" containerID="cri-o://5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" Sep 16 19:53:46.015504 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.015275 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" containerName="main" containerID="cri-o://5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" gracePeriod=60 Sep 16 19:53:46.383825 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.383795 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:46.465726 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465685 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.465926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.465926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465772 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.465926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.465926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465842 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnr8f\" (UniqueName: \"kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.465926 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.465887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm\") pod \"54691fbb-5642-4635-b208-5a63fc43d68e\" (UID: \"54691fbb-5642-4635-b208-5a63fc43d68e\") " Sep 16 19:53:46.466207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466051 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:46.466207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466065 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home" (OuterVolumeSpecName: "home") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:46.466207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466079 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache" (OuterVolumeSpecName: "model-cache") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:46.466380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466215 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.466380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466237 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.466380 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.466251 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.468366 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.468342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm" (OuterVolumeSpecName: "dshm") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:53:46.468366 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.468359 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f" (OuterVolumeSpecName: "kube-api-access-cnr8f") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "kube-api-access-cnr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:53:46.468513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.468342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54691fbb-5642-4635-b208-5a63fc43d68e" (UID: "54691fbb-5642-4635-b208-5a63fc43d68e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:53:46.567374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.567259 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54691fbb-5642-4635-b208-5a63fc43d68e-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.567374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.567318 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnr8f\" (UniqueName: \"kubernetes.io/projected/54691fbb-5642-4635-b208-5a63fc43d68e-kube-api-access-cnr8f\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.567374 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.567328 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54691fbb-5642-4635-b208-5a63fc43d68e-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:53:46.819340 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.819223 2565 generic.go:356] "Generic (PLEG): container finished" podID="54691fbb-5642-4635-b208-5a63fc43d68e" containerID="5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" exitCode=0 Sep 16 19:53:46.819340 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.819298 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" Sep 16 19:53:46.819340 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.819307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" event={"ID":"54691fbb-5642-4635-b208-5a63fc43d68e","Type":"ContainerDied","Data":"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d"} Sep 16 19:53:46.819633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.819367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s" event={"ID":"54691fbb-5642-4635-b208-5a63fc43d68e","Type":"ContainerDied","Data":"9fb9496bb74139d21cfc23e62ed08b4417d76cca2ea8b096b9343719606bcca2"} Sep 16 19:53:46.819633 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.819393 2565 scope.go:117] "RemoveContainer" containerID="5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" Sep 16 19:53:46.829623 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.829604 2565 scope.go:117] "RemoveContainer" containerID="5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" Sep 16 19:53:46.829911 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:53:46.829893 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d\": container with ID starting with 5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d not found: ID does not exist" containerID="5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d" Sep 16 19:53:46.829957 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.829922 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d"} err="failed to get container status \"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d\": rpc error: code = NotFound desc = could not find container \"5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d\": container with ID starting with 5ff28c89a7b914c29bced7c8904b808d16a148fb48c1b118c43a393b870e257d not found: ID does not exist" Sep 16 19:53:46.843411 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.843379 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:46.847482 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:46.847460 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hhq7s"] Sep 16 19:53:48.253523 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:53:48.253491 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" path="/var/lib/kubelet/pods/54691fbb-5642-4635-b208-5a63fc43d68e/volumes" Sep 16 19:54:03.452223 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.452186 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:54:03.452745 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.452634 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" containerName="main" Sep 16 19:54:03.452745 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.452647 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" containerName="main" Sep 16 19:54:03.452745 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.452708 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54691fbb-5642-4635-b208-5a63fc43d68e" containerName="main" Sep 16 19:54:03.456104 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.456080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.458486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.458463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\"" Sep 16 19:54:03.458619 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.458600 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\"" Sep 16 19:54:03.458680 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.458653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-frq67\"" Sep 16 19:54:03.458885 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.458868 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\"" Sep 16 19:54:03.459243 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.459224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\"" Sep 16 19:54:03.459612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.459595 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:54:03.469920 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.469897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:54:03.623524 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swlxz\" (UniqueName: \"kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623732 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623947 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623947 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.623947 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.623820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.724753 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.724709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.724989 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.724812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swlxz\" (UniqueName: \"kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.724989 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.724949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725115 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.724991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725115 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725226 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725226 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725360 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725704 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.725992 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.726091 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.725987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.726351 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.726323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.726808 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.726781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.729250 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.729224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.729535 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.729516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.742905 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.742878 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swlxz\" (UniqueName: \"kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:03.766958 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:03.766930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:04.100981 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:04.100953 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:54:04.102260 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:54:04.102239 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48346a75_d221_43fe_977e_49d0ac3f8c58.slice/crio-cd22c4a67b58e2a9b27f6d423a81d16847a13f22d4ab81407923feb97a896f42 WatchSource:0}: Error finding container cd22c4a67b58e2a9b27f6d423a81d16847a13f22d4ab81407923feb97a896f42: Status 404 returned error can't find the container with id cd22c4a67b58e2a9b27f6d423a81d16847a13f22d4ab81407923feb97a896f42 Sep 16 19:54:04.928023 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:04.927980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerStarted","Data":"8d2660b83ef79568d316820945be35acd08f27a8679849152f2d871d8fe11ca5"} Sep 16 19:54:04.928456 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:04.928058 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerStarted","Data":"cd22c4a67b58e2a9b27f6d423a81d16847a13f22d4ab81407923feb97a896f42"} Sep 16 19:54:06.943389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:06.943300 2565 generic.go:356] "Generic (PLEG): container finished" podID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerID="8d2660b83ef79568d316820945be35acd08f27a8679849152f2d871d8fe11ca5" exitCode=0 Sep 16 19:54:06.943757 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:06.943383 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerDied","Data":"8d2660b83ef79568d316820945be35acd08f27a8679849152f2d871d8fe11ca5"} Sep 16 19:54:07.953207 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:07.953165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerStarted","Data":"7ac43511e3c81b1dae300831e43fe16b4b3eea04b4d0ba0dcfaea200adb1ace4"} Sep 16 19:54:07.985574 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:07.985508 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podStartSLOduration=2.598929155 podStartE2EDuration="4.985485774s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.386556619s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:07.980409672 +0000 UTC m=+2932.186365425" watchObservedRunningTime="2026-09-16 19:54:07.985485774 +0000 UTC m=+2932.191441529" Sep 16 19:54:08.767800 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:08.767757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:13.767931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:13.767887 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:54:13.769787 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:13.769757 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:54:23.767927 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:23.767879 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:54:33.767977 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:33.767921 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:54:34.110754 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:34.110667 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:54:42.117378 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:42.117262 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:54:43.768032 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:43.767989 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:54:49.131417 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:49.131346 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" containerID="cri-o://66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6" gracePeriod=45 Sep 16 19:54:52.117228 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:52.117188 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:54:53.767653 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:54:53.767610 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:02.117119 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:02.117081 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:55:02.117621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:02.117212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:55:03.767531 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:03.767489 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:12.117146 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:12.117104 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:55:13.767684 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:13.767640 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:19.391518 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.391447 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:55:19.393510 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.393483 2565 generic.go:356] "Generic (PLEG): container finished" podID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerID="66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6" exitCode=0 Sep 16 19:55:19.393646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.393530 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" Sep 16 19:55:19.393646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.393561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" event={"ID":"7a5b51fb-ddf3-436b-a899-a60617ff128e","Type":"ContainerDied","Data":"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6"} Sep 16 19:55:19.393646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.393616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg" event={"ID":"7a5b51fb-ddf3-436b-a899-a60617ff128e","Type":"ContainerDied","Data":"b611ec3baa263889917cc94fd2f7a083371ced91f4c98c3b464c44135576a88c"} Sep 16 19:55:19.393646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.393644 2565 scope.go:117] "RemoveContainer" containerID="66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6" Sep 16 19:55:19.402854 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.402836 2565 scope.go:117] "RemoveContainer" containerID="66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6" Sep 16 19:55:19.403153 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:55:19.403133 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6\": container with ID starting with 66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6 not found: ID does not exist" containerID="66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6" Sep 16 19:55:19.403232 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.403163 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6"} err="failed to get container status \"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6\": rpc error: code = NotFound desc = could not find container \"66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6\": container with ID starting with 66bbe7685b84e1ed5316f1cf82227fc915c3b43bdb1336b1b7158339d2986be6 not found: ID does not exist" Sep 16 19:55:19.421604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.421573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs\") pod \"7a5b51fb-ddf3-436b-a899-a60617ff128e\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " Sep 16 19:55:19.421604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.421609 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzvw5\" (UniqueName: \"kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5\") pod \"7a5b51fb-ddf3-436b-a899-a60617ff128e\" (UID: \"7a5b51fb-ddf3-436b-a899-a60617ff128e\") " Sep 16 19:55:19.423872 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.423838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a5b51fb-ddf3-436b-a899-a60617ff128e" (UID: "7a5b51fb-ddf3-436b-a899-a60617ff128e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:55:19.423974 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.423898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5" (OuterVolumeSpecName: "kube-api-access-rzvw5") pod "7a5b51fb-ddf3-436b-a899-a60617ff128e" (UID: "7a5b51fb-ddf3-436b-a899-a60617ff128e"). InnerVolumeSpecName "kube-api-access-rzvw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:55:19.522943 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.522905 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a5b51fb-ddf3-436b-a899-a60617ff128e-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:55:19.522943 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.522936 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvw5\" (UniqueName: \"kubernetes.io/projected/7a5b51fb-ddf3-436b-a899-a60617ff128e-kube-api-access-rzvw5\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:55:19.720522 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.720487 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:55:19.722711 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:19.722689 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schefbqjg"] Sep 16 19:55:20.253451 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:20.253415 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" path="/var/lib/kubelet/pods/7a5b51fb-ddf3-436b-a899-a60617ff128e/volumes" Sep 16 19:55:23.768195 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:23.768157 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:29.302389 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.302354 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:55:29.302777 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.302764 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" Sep 16 19:55:29.302777 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.302778 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" Sep 16 19:55:29.302857 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.302852 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a5b51fb-ddf3-436b-a899-a60617ff128e" containerName="main" Sep 16 19:55:29.306106 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.306082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.309244 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.309221 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\"" Sep 16 19:55:29.310384 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.310361 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\"" Sep 16 19:55:29.310501 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.310478 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\"" Sep 16 19:55:29.310595 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.310420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-b6d42\"" Sep 16 19:55:29.310674 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.310425 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-dfc2t\"" Sep 16 19:55:29.319802 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.319772 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:55:29.416932 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.416892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s7g\" (UniqueName: \"kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.417124 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.417006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.518519 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.518472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.518735 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.518576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s7g\" (UniqueName: \"kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.521878 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.521847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.529212 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.529174 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s7g\" (UniqueName: \"kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.621491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.621391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:29.913882 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:29.913848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:55:29.914774 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:55:29.914733 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd368c2ca_684b_410e_97bd_f61506029cbd.slice/crio-81567dd0e04c78b9b20bdd7785728171d1dd358c60cdd8b4b40b1b5e9e0455ea WatchSource:0}: Error finding container 81567dd0e04c78b9b20bdd7785728171d1dd358c60cdd8b4b40b1b5e9e0455ea: Status 404 returned error can't find the container with id 81567dd0e04c78b9b20bdd7785728171d1dd358c60cdd8b4b40b1b5e9e0455ea Sep 16 19:55:30.463303 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:30.463243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" event={"ID":"d368c2ca-684b-410e-97bd-f61506029cbd","Type":"ContainerStarted","Data":"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a"} Sep 16 19:55:30.463781 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:30.463314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" event={"ID":"d368c2ca-684b-410e-97bd-f61506029cbd","Type":"ContainerStarted","Data":"81567dd0e04c78b9b20bdd7785728171d1dd358c60cdd8b4b40b1b5e9e0455ea"} Sep 16 19:55:30.463781 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:30.463437 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:55:30.485629 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:30.485572 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podStartSLOduration=1.4855556939999999 podStartE2EDuration="1.485555694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:30.482455609 +0000 UTC m=+3014.688411362" watchObservedRunningTime="2026-09-16 19:55:30.485555694 +0000 UTC m=+3014.691511446" Sep 16 19:55:33.767696 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:33.767649 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:43.767755 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:43.767706 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:55:53.768461 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:55:53.768401 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:56:01.472050 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:01.472020 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:56:03.778328 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:03.778277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:56:03.786368 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:03.786336 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:56:09.113161 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:09.113068 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:56:24.136559 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:24.136510 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" containerID="cri-o://7ac43511e3c81b1dae300831e43fe16b4b3eea04b4d0ba0dcfaea200adb1ace4" gracePeriod=45 Sep 16 19:56:24.779412 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:24.779373 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused" Sep 16 19:56:24.791338 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:24.791268 2565 generic.go:356] "Generic (PLEG): container finished" podID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerID="7ac43511e3c81b1dae300831e43fe16b4b3eea04b4d0ba0dcfaea200adb1ace4" exitCode=0 Sep 16 19:56:24.791512 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:24.791329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerDied","Data":"7ac43511e3c81b1dae300831e43fe16b4b3eea04b4d0ba0dcfaea200adb1ace4"} Sep 16 19:56:24.900516 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:24.900492 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:56:25.030060 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030022 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swlxz\" (UniqueName: \"kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030077 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030165 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030246 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030219 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030248 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030494 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030339 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home\") pod \"48346a75-d221-43fe-977e-49d0ac3f8c58\" (UID: \"48346a75-d221-43fe-977e-49d0ac3f8c58\") " Sep 16 19:56:25.030883 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.030807 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:56:25.031100 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.031078 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache" (OuterVolumeSpecName: "model-cache") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:56:25.031396 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.031370 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home" (OuterVolumeSpecName: "home") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:56:25.032698 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.032672 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm" (OuterVolumeSpecName: "dshm") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:56:25.032805 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.032722 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz" (OuterVolumeSpecName: "kube-api-access-swlxz") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "kube-api-access-swlxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:56:25.032869 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.032826 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:56:25.045931 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.045903 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:56:25.049370 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.049339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48346a75-d221-43fe-977e-49d0ac3f8c58" (UID: "48346a75-d221-43fe-977e-49d0ac3f8c58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:56:25.131966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131932 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.131966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131963 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48346a75-d221-43fe-977e-49d0ac3f8c58-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.131966 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131972 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.132177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131981 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swlxz\" (UniqueName: \"kubernetes.io/projected/48346a75-d221-43fe-977e-49d0ac3f8c58-kube-api-access-swlxz\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.132177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131991 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346a75-d221-43fe-977e-49d0ac3f8c58-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.132177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.131999 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.132177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.132006 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.132177 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.132014 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48346a75-d221-43fe-977e-49d0ac3f8c58-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:56:25.799210 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.799183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" Sep 16 19:56:25.799686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.799207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8" event={"ID":"48346a75-d221-43fe-977e-49d0ac3f8c58","Type":"ContainerDied","Data":"cd22c4a67b58e2a9b27f6d423a81d16847a13f22d4ab81407923feb97a896f42"} Sep 16 19:56:25.799686 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.799276 2565 scope.go:117] "RemoveContainer" containerID="7ac43511e3c81b1dae300831e43fe16b4b3eea04b4d0ba0dcfaea200adb1ace4" Sep 16 19:56:25.809548 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.809528 2565 scope.go:117] "RemoveContainer" containerID="8d2660b83ef79568d316820945be35acd08f27a8679849152f2d871d8fe11ca5" Sep 16 19:56:25.824111 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.824084 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:56:25.830513 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:25.830489 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4897788f9g2z8"] Sep 16 19:56:26.253430 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:56:26.253396 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" path="/var/lib/kubelet/pods/48346a75-d221-43fe-977e-49d0ac3f8c58/volumes" Sep 16 19:57:40.662452 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662360 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 19:57:40.663065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662777 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" Sep 16 19:57:40.663065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662791 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" Sep 16 19:57:40.663065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662822 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="storage-initializer" Sep 16 19:57:40.663065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662828 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="storage-initializer" Sep 16 19:57:40.663065 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.662896 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48346a75-d221-43fe-977e-49d0ac3f8c58" containerName="main" Sep 16 19:57:40.666299 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.666265 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.669491 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.669473 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\"" Sep 16 19:57:40.669604 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.669580 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\"" Sep 16 19:57:40.670011 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.669992 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\"" Sep 16 19:57:40.670103 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.670064 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-gslfn\"" Sep 16 19:57:40.681104 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.681078 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 19:57:40.705937 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.705899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706230 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tgsh\" (UniqueName: \"kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706356 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706424 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.706540 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.706530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807359 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tgsh\" (UniqueName: \"kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807612 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.807891 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.808005 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.807907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.809964 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.809946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.810519 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.810500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.816143 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.816119 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tgsh\" (UniqueName: \"kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:40.976930 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:40.976895 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:57:41.261849 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:41.261819 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 19:57:41.306427 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:41.306403 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:57:42.229377 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:42.229339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerStarted","Data":"672542e006102a7672aeaba332ff206b97ff110e1e2ea4ef020111cb2fc05c94"} Sep 16 19:57:47.975682 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:47.975645 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:57:48.275289 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:48.275241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerStarted","Data":"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010"} Sep 16 19:57:50.288475 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:50.288441 2565 generic.go:356] "Generic (PLEG): container finished" podID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerID="ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010" exitCode=0 Sep 16 19:57:50.288864 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:50.288519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerDied","Data":"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010"} Sep 16 19:57:51.296788 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:51.296751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerStarted","Data":"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58"} Sep 16 19:57:51.297194 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:51.296815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerStarted","Data":"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95"} Sep 16 19:57:51.336928 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:51.336874 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podStartSLOduration=3.15821538 podStartE2EDuration="11.336856753s" totalImagesPullingTime="6.892681372s" totalInitContainerRuntime="1.285960001s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:51.335348885 +0000 UTC m=+3155.541304638" watchObservedRunningTime="2026-09-16 19:57:51.336856753 +0000 UTC m=+3155.542812505" Sep 16 19:57:51.472743 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:51.472698 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:57:51.977960 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:57:51.977933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:58:00.977478 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:00.977436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:58:00.979142 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:00.979110 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:01.472486 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:01.472438 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:58:03.096321 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:03.096259 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" containerID="cri-o://627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a" gracePeriod=45 Sep 16 19:58:10.977543 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:10.977490 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:11.472464 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:11.472421 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:58:11.472657 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:11.472536 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:58:20.977701 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:20.977656 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:21.472440 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:21.472393 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:58:30.977829 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:30.977777 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:31.472731 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:31.472689 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 19:58:33.355078 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.355008 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:58:33.418534 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.418501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs\") pod \"d368c2ca-684b-410e-97bd-f61506029cbd\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " Sep 16 19:58:33.418746 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.418577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8s7g\" (UniqueName: \"kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g\") pod \"d368c2ca-684b-410e-97bd-f61506029cbd\" (UID: \"d368c2ca-684b-410e-97bd-f61506029cbd\") " Sep 16 19:58:33.420959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.420925 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d368c2ca-684b-410e-97bd-f61506029cbd" (UID: "d368c2ca-684b-410e-97bd-f61506029cbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:58:33.420959 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.420934 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g" (OuterVolumeSpecName: "kube-api-access-b8s7g") pod "d368c2ca-684b-410e-97bd-f61506029cbd" (UID: "d368c2ca-684b-410e-97bd-f61506029cbd"). InnerVolumeSpecName "kube-api-access-b8s7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:58:33.519766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.519732 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d368c2ca-684b-410e-97bd-f61506029cbd-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:58:33.519766 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.519763 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8s7g\" (UniqueName: \"kubernetes.io/projected/d368c2ca-684b-410e-97bd-f61506029cbd-kube-api-access-b8s7g\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 19:58:33.547713 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.547682 2565 generic.go:356] "Generic (PLEG): container finished" podID="d368c2ca-684b-410e-97bd-f61506029cbd" containerID="627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a" exitCode=0 Sep 16 19:58:33.547915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.547730 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" Sep 16 19:58:33.547915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.547776 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" event={"ID":"d368c2ca-684b-410e-97bd-f61506029cbd","Type":"ContainerDied","Data":"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a"} Sep 16 19:58:33.547915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.547835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6" event={"ID":"d368c2ca-684b-410e-97bd-f61506029cbd","Type":"ContainerDied","Data":"81567dd0e04c78b9b20bdd7785728171d1dd358c60cdd8b4b40b1b5e9e0455ea"} Sep 16 19:58:33.547915 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.547864 2565 scope.go:117] "RemoveContainer" containerID="627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a" Sep 16 19:58:33.559086 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.559066 2565 scope.go:117] "RemoveContainer" containerID="627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a" Sep 16 19:58:33.559417 ip-10-0-128-148 kubenswrapper[2565]: E0916 19:58:33.559399 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a\": container with ID starting with 627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a not found: ID does not exist" containerID="627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a" Sep 16 19:58:33.559475 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.559427 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a"} err="failed to get container status \"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a\": rpc error: code = NotFound desc = could not find container \"627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a\": container with ID starting with 627bd87002aa751288a8538c353b567c4fb57049f309fbd23eee956b9110b42a not found: ID does not exist" Sep 16 19:58:33.572676 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.572638 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:58:33.579448 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:33.579414 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegznn6"] Sep 16 19:58:34.254761 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:34.254721 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" path="/var/lib/kubelet/pods/d368c2ca-684b-410e-97bd-f61506029cbd/volumes" Sep 16 19:58:40.977750 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:40.977706 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:50.977783 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:50.977720 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:58:53.788714 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.788676 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 19:58:53.789343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.789124 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" Sep 16 19:58:53.789343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.789136 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" Sep 16 19:58:53.789343 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.789197 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d368c2ca-684b-410e-97bd-f61506029cbd" containerName="main" Sep 16 19:58:53.793682 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.793658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:53.796454 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.796415 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-zx49k\"" Sep 16 19:58:53.796585 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.796457 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-6pmcr\"" Sep 16 19:58:53.796585 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.796542 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\"" Sep 16 19:58:53.796724 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.796677 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\"" Sep 16 19:58:53.796809 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.796788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\"" Sep 16 19:58:53.803776 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.803746 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 19:58:53.914908 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.914863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkk2\" (UniqueName: \"kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:53.915092 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:53.915028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.015898 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.015854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.016066 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.015967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkk2\" (UniqueName: \"kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.018551 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.018526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.025855 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.025825 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkk2\" (UniqueName: \"kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.106725 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.106629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.388191 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.388118 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 19:58:54.389094 ip-10-0-128-148 kubenswrapper[2565]: W0916 19:58:54.389063 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f834a2d_80ac_4521_abc2_2711f6a532bc.slice/crio-05b44277ebd2be16676cc62b664dedead615048abfb0cd1ba66f2af015e3196c WatchSource:0}: Error finding container 05b44277ebd2be16676cc62b664dedead615048abfb0cd1ba66f2af015e3196c: Status 404 returned error can't find the container with id 05b44277ebd2be16676cc62b664dedead615048abfb0cd1ba66f2af015e3196c Sep 16 19:58:54.678431 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.678320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" event={"ID":"5f834a2d-80ac-4521-abc2-2711f6a532bc","Type":"ContainerStarted","Data":"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9"} Sep 16 19:58:54.678431 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.678389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" event={"ID":"5f834a2d-80ac-4521-abc2-2711f6a532bc","Type":"ContainerStarted","Data":"05b44277ebd2be16676cc62b664dedead615048abfb0cd1ba66f2af015e3196c"} Sep 16 19:58:54.678646 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.678499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:58:54.695667 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:58:54.695615 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podStartSLOduration=1.6955991 podStartE2EDuration="1.6955991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:54.694367714 +0000 UTC m=+3218.900323474" watchObservedRunningTime="2026-09-16 19:58:54.6955991 +0000 UTC m=+3218.901554852" Sep 16 19:59:00.977710 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:00.977664 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:59:10.977376 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:10.977264 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:59:20.978103 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:20.978047 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:59:25.685689 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:25.685657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 19:59:30.977675 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:30.977621 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:59:40.978206 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:40.978154 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused" Sep 16 19:59:50.987621 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:50.987585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:59:50.995455 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:50.995428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 19:59:56.114996 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:56.113875 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 19:59:56.115832 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:56.115037 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar" containerID="cri-o://e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58" gracePeriod=60 Sep 16 19:59:57.046363 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:57.046333 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp_19115b96-95ba-4bf3-94c8-d0d9fac687a5/modelcar/0.log" Sep 16 19:59:57.047131 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:57.047107 2565 generic.go:356] "Generic (PLEG): container finished" podID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerID="e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58" exitCode=143 Sep 16 19:59:57.047247 ip-10-0-128-148 kubenswrapper[2565]: I0916 19:59:57.047193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerDied","Data":"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58"} Sep 16 20:00:11.148606 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:11.148560 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" containerID="cri-o://f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95" gracePeriod=45 Sep 16 20:00:12.011835 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.011811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp_19115b96-95ba-4bf3-94c8-d0d9fac687a5/modelcar/0.log" Sep 16 20:00:12.014792 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.014771 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 20:00:12.134718 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.134688 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp_19115b96-95ba-4bf3-94c8-d0d9fac687a5/modelcar/0.log" Sep 16 20:00:12.135398 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.135371 2565 generic.go:356] "Generic (PLEG): container finished" podID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerID="f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95" exitCode=0 Sep 16 20:00:12.135518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.135422 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" Sep 16 20:00:12.135518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.135423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerDied","Data":"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95"} Sep 16 20:00:12.135518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.135490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" event={"ID":"19115b96-95ba-4bf3-94c8-d0d9fac687a5","Type":"ContainerDied","Data":"672542e006102a7672aeaba332ff206b97ff110e1e2ea4ef020111cb2fc05c94"} Sep 16 20:00:12.135645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.135521 2565 scope.go:117] "RemoveContainer" containerID="e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58" Sep 16 20:00:12.142331 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142406 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142631 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142631 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.142850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142829 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tgsh\" (UniqueName: \"kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh\") pod \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\" (UID: \"19115b96-95ba-4bf3-94c8-d0d9fac687a5\") " Sep 16 20:00:12.143135 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.142838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:00:12.143135 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.143047 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home" (OuterVolumeSpecName: "home") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:00:12.143300 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.143085 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:00:12.143661 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.143640 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.143713 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.143668 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.143713 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.143683 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.145162 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.145140 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh" (OuterVolumeSpecName: "kube-api-access-8tgsh") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "kube-api-access-8tgsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:00:12.145341 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.145316 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:00:12.145438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.145344 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm" (OuterVolumeSpecName: "dshm") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:00:12.145632 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.145616 2565 scope.go:117] "RemoveContainer" containerID="f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95" Sep 16 20:00:12.155608 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.155581 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19115b96-95ba-4bf3-94c8-d0d9fac687a5" (UID: "19115b96-95ba-4bf3-94c8-d0d9fac687a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:00:12.157789 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.157687 2565 scope.go:117] "RemoveContainer" containerID="ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010" Sep 16 20:00:12.168181 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.168161 2565 scope.go:117] "RemoveContainer" containerID="e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58" Sep 16 20:00:12.168668 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:00:12.168646 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58\": container with ID starting with e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58 not found: ID does not exist" containerID="e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58" Sep 16 20:00:12.168749 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.168676 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58"} err="failed to get container status \"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58\": rpc error: code = NotFound desc = could not find container \"e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58\": container with ID starting with e936496f8bfbf3c3e4b5917d5352580add5ad2fd0c9b614b4e335128df126a58 not found: ID does not exist" Sep 16 20:00:12.168749 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.168694 2565 scope.go:117] "RemoveContainer" containerID="f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95" Sep 16 20:00:12.169001 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:00:12.168983 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95\": container with ID starting with f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95 not found: ID does not exist" containerID="f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95" Sep 16 20:00:12.169039 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.169009 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95"} err="failed to get container status \"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95\": rpc error: code = NotFound desc = could not find container \"f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95\": container with ID starting with f64748eb04c76b2431df6b437eb124fadad580de8c7e821a81cec7abb7010e95 not found: ID does not exist" Sep 16 20:00:12.169039 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.169025 2565 scope.go:117] "RemoveContainer" containerID="ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010" Sep 16 20:00:12.169260 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:00:12.169245 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010\": container with ID starting with ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010 not found: ID does not exist" containerID="ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010" Sep 16 20:00:12.169330 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.169264 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010"} err="failed to get container status \"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010\": rpc error: code = NotFound desc = could not find container \"ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010\": container with ID starting with ea58ca2a753cdc78d6270d5669cf5c9da912e1a9ed1003f099182788249a0010 not found: ID does not exist" Sep 16 20:00:12.244941 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.244850 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.244941 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.244883 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.244941 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.244894 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19115b96-95ba-4bf3-94c8-d0d9fac687a5-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.244941 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.244909 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tgsh\" (UniqueName: \"kubernetes.io/projected/19115b96-95ba-4bf3-94c8-d0d9fac687a5-kube-api-access-8tgsh\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:00:12.454106 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.454067 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 20:00:12.459556 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.459528 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp"] Sep 16 20:00:12.988878 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:12.988824 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-64d87fcdc7g45mp" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Sep 16 20:00:14.258793 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:14.258752 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" path="/var/lib/kubelet/pods/19115b96-95ba-4bf3-94c8-d0d9fac687a5/volumes" Sep 16 20:00:15.687420 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:15.687379 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:00:15.741965 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:15.741920 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 20:00:25.686226 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:25.686187 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:00:30.763570 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:30.763500 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" containerID="cri-o://b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9" gracePeriod=45 Sep 16 20:00:35.686029 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:35.685983 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:00:35.686522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:35.686109 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 20:00:45.686059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:45.685962 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:00:54.058402 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.058365 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:00:54.058985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.058965 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar" Sep 16 20:00:54.059084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.058989 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar" Sep 16 20:00:54.059084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059023 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" Sep 16 20:00:54.059084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059032 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" Sep 16 20:00:54.059084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059046 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar-init" Sep 16 20:00:54.059084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059056 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar-init" Sep 16 20:00:54.059368 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059151 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="main" Sep 16 20:00:54.059368 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.059168 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19115b96-95ba-4bf3-94c8-d0d9fac687a5" containerName="modelcar" Sep 16 20:00:54.063531 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.063509 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.066238 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.066215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Sep 16 20:00:54.066399 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.066311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\"" Sep 16 20:00:54.066399 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.066315 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\"" Sep 16 20:00:54.066399 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.066351 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:00:54.066399 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.066317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\"" Sep 16 20:00:54.067081 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.067065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-j7nb9\"" Sep 16 20:00:54.078472 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.078445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:00:54.102713 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102713 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102940 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6dz\" (UniqueName: \"kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102940 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102940 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102940 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.102940 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.103166 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.102979 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.203833 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.203833 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6dz\" (UniqueName: \"kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.203998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204445 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204445 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204520 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204520 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.204706 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.204688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.206534 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.206499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.206721 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.206705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.216079 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.216056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6dz\" (UniqueName: \"kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz\") pod \"custom-route-timeout-test-kserve-85f45479b-jx2d9\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.375062 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.374957 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:00:54.661102 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:54.661071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:00:54.662758 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:00:54.662721 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0514d5_b130_4d41_a68d_357faa2abb1a.slice/crio-d33021c810940516031c36627ce897397e867be604fb6b337a7ed5406a1df090 WatchSource:0}: Error finding container d33021c810940516031c36627ce897397e867be604fb6b337a7ed5406a1df090: Status 404 returned error can't find the container with id d33021c810940516031c36627ce897397e867be604fb6b337a7ed5406a1df090 Sep 16 20:00:55.403746 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:55.403708 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerStarted","Data":"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e"} Sep 16 20:00:55.404258 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:55.403760 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerStarted","Data":"d33021c810940516031c36627ce897397e867be604fb6b337a7ed5406a1df090"} Sep 16 20:00:55.686942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:55.686845 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:00:57.420364 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:57.420255 2565 generic.go:356] "Generic (PLEG): container finished" podID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerID="b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e" exitCode=0 Sep 16 20:00:57.420364 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:57.420308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerDied","Data":"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e"} Sep 16 20:00:58.428767 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:58.428730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerStarted","Data":"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145"} Sep 16 20:00:58.452095 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:58.452034 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podStartSLOduration=2.082142677 podStartE2EDuration="4.452015035s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.369872358s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:58.450727194 +0000 UTC m=+3342.656682943" watchObservedRunningTime="2026-09-16 20:00:58.452015035 +0000 UTC m=+3342.657970791" Sep 16 20:00:59.375404 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:00:59.375367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:01:01.024570 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.024508 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 20:01:01.072997 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.072961 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trkk2\" (UniqueName: \"kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2\") pod \"5f834a2d-80ac-4521-abc2-2711f6a532bc\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " Sep 16 20:01:01.073181 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.073046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs\") pod \"5f834a2d-80ac-4521-abc2-2711f6a532bc\" (UID: \"5f834a2d-80ac-4521-abc2-2711f6a532bc\") " Sep 16 20:01:01.075529 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.075496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2" (OuterVolumeSpecName: "kube-api-access-trkk2") pod "5f834a2d-80ac-4521-abc2-2711f6a532bc" (UID: "5f834a2d-80ac-4521-abc2-2711f6a532bc"). InnerVolumeSpecName "kube-api-access-trkk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:01:01.075529 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.075520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f834a2d-80ac-4521-abc2-2711f6a532bc" (UID: "5f834a2d-80ac-4521-abc2-2711f6a532bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:01:01.173986 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.173945 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkk2\" (UniqueName: \"kubernetes.io/projected/5f834a2d-80ac-4521-abc2-2711f6a532bc-kube-api-access-trkk2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:01:01.173986 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.173977 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f834a2d-80ac-4521-abc2-2711f6a532bc-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:01:01.447559 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.447458 2565 generic.go:356] "Generic (PLEG): container finished" podID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerID="b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9" exitCode=0 Sep 16 20:01:01.447559 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.447499 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" Sep 16 20:01:01.447765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.447554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" event={"ID":"5f834a2d-80ac-4521-abc2-2711f6a532bc","Type":"ContainerDied","Data":"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9"} Sep 16 20:01:01.447765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.447609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9" event={"ID":"5f834a2d-80ac-4521-abc2-2711f6a532bc","Type":"ContainerDied","Data":"05b44277ebd2be16676cc62b664dedead615048abfb0cd1ba66f2af015e3196c"} Sep 16 20:01:01.447765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.447633 2565 scope.go:117] "RemoveContainer" containerID="b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9" Sep 16 20:01:01.457926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.457900 2565 scope.go:117] "RemoveContainer" containerID="b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9" Sep 16 20:01:01.458249 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:01.458230 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9\": container with ID starting with b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9 not found: ID does not exist" containerID="b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9" Sep 16 20:01:01.458344 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.458260 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9"} err="failed to get container status \"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9\": rpc error: code = NotFound desc = could not find container \"b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9\": container with ID starting with b0cba6cb488db1bcd3770cfcdae2f4e59e46b879cdb8f80189d241ccb2cfdad9 not found: ID does not exist" Sep 16 20:01:01.471009 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.470983 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 20:01:01.479678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:01.479639 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2s4w9"] Sep 16 20:01:02.254515 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:02.254475 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" path="/var/lib/kubelet/pods/5f834a2d-80ac-4521-abc2-2711f6a532bc/volumes" Sep 16 20:01:04.375987 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:04.375948 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:01:04.377680 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:04.377643 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:14.375774 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:14.375732 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:16.241806 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.241770 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:01:16.242263 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.242248 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" Sep 16 20:01:16.242339 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.242264 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" Sep 16 20:01:16.242399 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.242387 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f834a2d-80ac-4521-abc2-2711f6a532bc" containerName="main" Sep 16 20:01:16.245814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.245786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.249711 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.249686 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-gfxzt\"" Sep 16 20:01:16.250092 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.249687 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-2fngv\"" Sep 16 20:01:16.250189 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.249755 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\"" Sep 16 20:01:16.250261 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.249833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\"" Sep 16 20:01:16.250412 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.249837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\"" Sep 16 20:01:16.255639 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.255611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:01:16.423121 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.423078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.423373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.423153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66f4\" (UniqueName: \"kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.524633 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.524528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.524633 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.524581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66f4\" (UniqueName: \"kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.527269 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.527243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.533724 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.533695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66f4\" (UniqueName: \"kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.560133 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.560096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:16.846304 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:16.846259 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:01:16.848225 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:01:16.848198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5202628f_f07c_4162_a464_7d8b7c6c37ff.slice/crio-7c7308572d55ee839b4c90fa5e077d96f27ea74f76bd0a46101887267b03ddd9 WatchSource:0}: Error finding container 7c7308572d55ee839b4c90fa5e077d96f27ea74f76bd0a46101887267b03ddd9: Status 404 returned error can't find the container with id 7c7308572d55ee839b4c90fa5e077d96f27ea74f76bd0a46101887267b03ddd9 Sep 16 20:01:17.559338 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:17.559275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" event={"ID":"5202628f-f07c-4162-a464-7d8b7c6c37ff","Type":"ContainerStarted","Data":"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf"} Sep 16 20:01:17.559754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:17.559362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" event={"ID":"5202628f-f07c-4162-a464-7d8b7c6c37ff","Type":"ContainerStarted","Data":"7c7308572d55ee839b4c90fa5e077d96f27ea74f76bd0a46101887267b03ddd9"} Sep 16 20:01:17.559754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:17.559491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:01:17.579782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:17.579681 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" podStartSLOduration=1.579657777 podStartE2EDuration="1.579657777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:17.576711037 +0000 UTC m=+3361.782666790" watchObservedRunningTime="2026-09-16 20:01:17.579657777 +0000 UTC m=+3361.785613538" Sep 16 20:01:18.643275 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:18.643219 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:18.643803 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:18.643357 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:19.14333444 +0000 UTC m=+3363.349290190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:18.659456 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:18.659422 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:01:19.153305 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:19.148343 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:19.153305 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:19.148450 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:20.148428616 +0000 UTC m=+3364.354384366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:20.156728 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:20.156691 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:20.157123 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:20.156777 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:22.156755368 +0000 UTC m=+3366.362711109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:22.176162 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:22.176113 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:22.176584 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:22.176203 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:26.176188414 +0000 UTC m=+3370.382144145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:24.375460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:24.375409 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:26.213745 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:26.213710 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:26.214155 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:26.213798 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:34.213779978 +0000 UTC m=+3378.419735710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:34.294091 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:34.294054 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:34.294567 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:34.294125 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:01:50.294110894 +0000 UTC m=+3394.500066625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:34.375758 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:34.375708 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:34.592109 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:34.592002 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" containerID="cri-o://c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf" gracePeriod=45 Sep 16 20:01:44.375574 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:44.375527 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:49.150296 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.150239 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:01:49.155216 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.155195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.160976 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.160948 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\"" Sep 16 20:01:49.161174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.161003 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qqkdg\"" Sep 16 20:01:49.161529 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.161236 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Sep 16 20:01:49.161529 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.161252 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:01:49.161698 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.161618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Sep 16 20:01:49.161750 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.161723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Sep 16 20:01:49.172258 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.172216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:01:49.339863 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.339815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfb5\" (UniqueName: \"kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.339863 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.339852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.339877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.339928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.339996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.340022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.340119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.340472 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.340188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.440955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfb5\" (UniqueName: \"kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441546 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441546 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441546 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441768 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441872 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.441945 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.441930 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.442261 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.442236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.443953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.443931 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.444128 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.444108 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.452947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.452922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfb5\" (UniqueName: \"kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5\") pod \"stop-feature-test-kserve-86c9894c98-j6cnt\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.466916 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.466890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:49.574254 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.574207 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:01:49.766063 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:49.766037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:01:49.767539 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:01:49.767503 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded09361f_0193_47dd_93ea_963367c14846.slice/crio-87387c20dcbf1498e04023c975114b52ea9be320ac9f9a4c1dbd8dea7ce04129 WatchSource:0}: Error finding container 87387c20dcbf1498e04023c975114b52ea9be320ac9f9a4c1dbd8dea7ce04129: Status 404 returned error can't find the container with id 87387c20dcbf1498e04023c975114b52ea9be320ac9f9a4c1dbd8dea7ce04129 Sep 16 20:01:50.350879 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:50.350839 2565 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:50.351419 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:01:50.350914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs podName:5202628f-f07c-4162-a464-7d8b7c6c37ff nodeName:}" failed. No retries permitted until 2026-09-16 20:02:22.350897772 +0000 UTC m=+3426.556853504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 16 20:01:50.763301 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:50.763248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerStarted","Data":"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47"} Sep 16 20:01:50.763479 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:50.763322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerStarted","Data":"87387c20dcbf1498e04023c975114b52ea9be320ac9f9a4c1dbd8dea7ce04129"} Sep 16 20:01:52.779385 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:52.779346 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed09361f-0193-47dd-93ea-963367c14846" containerID="3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47" exitCode=0 Sep 16 20:01:52.779767 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:52.779413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerDied","Data":"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47"} Sep 16 20:01:53.788854 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:53.788818 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerStarted","Data":"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d"} Sep 16 20:01:53.817664 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:53.817608 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podStartSLOduration=2.28515186 podStartE2EDuration="4.817590554s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.532438694s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:53.814171881 +0000 UTC m=+3398.020127666" watchObservedRunningTime="2026-09-16 20:01:53.817590554 +0000 UTC m=+3398.023546306" Sep 16 20:01:54.375904 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:54.375857 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:01:54.467196 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:54.467164 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:59.467506 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:59.467458 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:01:59.469299 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:59.469250 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:01:59.573770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:01:59.573727 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 16 20:02:04.376350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.376270 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:02:04.862155 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.862126 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:02:04.867114 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.867070 2565 generic.go:356] "Generic (PLEG): container finished" podID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerID="c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf" exitCode=0 Sep 16 20:02:04.867262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.867127 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" Sep 16 20:02:04.867262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.867152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" event={"ID":"5202628f-f07c-4162-a464-7d8b7c6c37ff","Type":"ContainerDied","Data":"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf"} Sep 16 20:02:04.867262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.867216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297" event={"ID":"5202628f-f07c-4162-a464-7d8b7c6c37ff","Type":"ContainerDied","Data":"7c7308572d55ee839b4c90fa5e077d96f27ea74f76bd0a46101887267b03ddd9"} Sep 16 20:02:04.867262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.867248 2565 scope.go:117] "RemoveContainer" containerID="c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf" Sep 16 20:02:04.877018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.877000 2565 scope.go:117] "RemoveContainer" containerID="c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf" Sep 16 20:02:04.877364 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:02:04.877345 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf\": container with ID starting with c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf not found: ID does not exist" containerID="c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf" Sep 16 20:02:04.877444 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.877375 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf"} err="failed to get container status \"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf\": rpc error: code = NotFound desc = could not find container \"c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf\": container with ID starting with c6833f13e08dbabd558041288636da8c75d83b022d4207c98fb93d1bb0cfa4cf not found: ID does not exist" Sep 16 20:02:04.899782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.899748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v66f4\" (UniqueName: \"kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4\") pod \"5202628f-f07c-4162-a464-7d8b7c6c37ff\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " Sep 16 20:02:04.899972 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.899902 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs\") pod \"5202628f-f07c-4162-a464-7d8b7c6c37ff\" (UID: \"5202628f-f07c-4162-a464-7d8b7c6c37ff\") " Sep 16 20:02:04.902493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.902460 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4" (OuterVolumeSpecName: "kube-api-access-v66f4") pod "5202628f-f07c-4162-a464-7d8b7c6c37ff" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff"). InnerVolumeSpecName "kube-api-access-v66f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:02:04.902650 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:04.902614 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5202628f-f07c-4162-a464-7d8b7c6c37ff" (UID: "5202628f-f07c-4162-a464-7d8b7c6c37ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:02:05.001131 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:05.001080 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5202628f-f07c-4162-a464-7d8b7c6c37ff-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:02:05.001131 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:05.001128 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66f4\" (UniqueName: \"kubernetes.io/projected/5202628f-f07c-4162-a464-7d8b7c6c37ff-kube-api-access-v66f4\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:02:05.191709 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:05.191667 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:02:05.198053 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:05.198021 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-66f5dd57f5-fj297"] Sep 16 20:02:06.256421 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:06.256379 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" path="/var/lib/kubelet/pods/5202628f-f07c-4162-a464-7d8b7c6c37ff/volumes" Sep 16 20:02:09.467677 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:09.467561 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:02:14.375822 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:14.375762 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:02:19.468021 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:19.467971 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:02:24.376050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:24.375995 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:02:29.467827 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:29.467773 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:02:34.375634 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:34.375584 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:02:39.468426 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:39.468374 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:02:44.375874 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:44.375822 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:02:49.468080 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:49.468023 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:02:54.386908 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:54.386873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:02:54.394729 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:54.394701 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:02:58.949303 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:58.949253 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:02:59.467763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:02:59.467717 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:03:09.467406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:09.467350 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:03:13.973005 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:13.972942 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" containerID="cri-o://53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145" gracePeriod=45 Sep 16 20:03:14.388593 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:14.388484 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused" Sep 16 20:03:15.041493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.041461 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:03:15.100441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100380 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100421 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100453 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100571 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100598 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq6dz\" (UniqueName: \"kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.100754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100632 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache\") pod \"ae0514d5-b130-4d41-a68d-357faa2abb1a\" (UID: \"ae0514d5-b130-4d41-a68d-357faa2abb1a\") " Sep 16 20:03:15.101058 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100817 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:03:15.101058 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.100954 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae0514d5-b130-4d41-a68d-357faa2abb1a-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.101058 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.101042 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:03:15.101264 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.101229 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home" (OuterVolumeSpecName: "home") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:03:15.103339 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.103302 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm" (OuterVolumeSpecName: "dshm") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:03:15.103505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.103476 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:03:15.103773 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.103750 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz" (OuterVolumeSpecName: "kube-api-access-bq6dz") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "kube-api-access-bq6dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:03:15.108040 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.108012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:03:15.113251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.113224 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae0514d5-b130-4d41-a68d-357faa2abb1a" (UID: "ae0514d5-b130-4d41-a68d-357faa2abb1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:03:15.201994 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.201956 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.201994 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.201987 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0514d5-b130-4d41-a68d-357faa2abb1a-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.201994 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.201997 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6dz\" (UniqueName: \"kubernetes.io/projected/ae0514d5-b130-4d41-a68d-357faa2abb1a-kube-api-access-bq6dz\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.202249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.202006 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.202249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.202015 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.202249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.202023 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.202249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.202031 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0514d5-b130-4d41-a68d-357faa2abb1a-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:03:15.337173 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.337133 2565 generic.go:356] "Generic (PLEG): container finished" podID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerID="53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145" exitCode=0 Sep 16 20:03:15.337373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.337189 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" Sep 16 20:03:15.337373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.337231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerDied","Data":"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145"} Sep 16 20:03:15.337373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.337309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9" event={"ID":"ae0514d5-b130-4d41-a68d-357faa2abb1a","Type":"ContainerDied","Data":"d33021c810940516031c36627ce897397e867be604fb6b337a7ed5406a1df090"} Sep 16 20:03:15.337373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.337334 2565 scope.go:117] "RemoveContainer" containerID="53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145" Sep 16 20:03:15.347581 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.347560 2565 scope.go:117] "RemoveContainer" containerID="b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e" Sep 16 20:03:15.356309 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.356269 2565 scope.go:117] "RemoveContainer" containerID="53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145" Sep 16 20:03:15.356638 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:03:15.356620 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145\": container with ID starting with 53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145 not found: ID does not exist" containerID="53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145" Sep 16 20:03:15.356703 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.356646 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145"} err="failed to get container status \"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145\": rpc error: code = NotFound desc = could not find container \"53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145\": container with ID starting with 53530fe2b4c1254bb31e2aad4bc7244870b76e3d99aa5a5e6bac6f0c1d996145 not found: ID does not exist" Sep 16 20:03:15.356703 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.356662 2565 scope.go:117] "RemoveContainer" containerID="b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e" Sep 16 20:03:15.356970 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:03:15.356946 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e\": container with ID starting with b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e not found: ID does not exist" containerID="b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e" Sep 16 20:03:15.357250 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.356977 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e"} err="failed to get container status \"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e\": rpc error: code = NotFound desc = could not find container \"b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e\": container with ID starting with b324512f89ac960366c453008dd08c53101bdb122e83036b597ee2c68d246b5e not found: ID does not exist" Sep 16 20:03:15.360550 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.360528 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:03:15.365868 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:15.365847 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f45479b-jx2d9"] Sep 16 20:03:16.253587 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:16.253550 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" path="/var/lib/kubelet/pods/ae0514d5-b130-4d41-a68d-357faa2abb1a/volumes" Sep 16 20:03:19.467702 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:19.467656 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:03:29.467596 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:29.467543 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:03:39.467885 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:39.467796 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:03:49.478613 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:49.478580 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:03:49.488473 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:03:49.488449 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:04:05.951050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951010 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:04:05.951553 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951536 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="storage-initializer" Sep 16 20:04:05.951607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951556 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="storage-initializer" Sep 16 20:04:05.951607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951582 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" Sep 16 20:04:05.951607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951588 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" Sep 16 20:04:05.951607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951598 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" Sep 16 20:04:05.951607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951604 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" Sep 16 20:04:05.951777 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951661 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae0514d5-b130-4d41-a68d-357faa2abb1a" containerName="main" Sep 16 20:04:05.951777 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.951669 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5202628f-f07c-4162-a464-7d8b7c6c37ff" containerName="main" Sep 16 20:04:05.954866 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.954848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:05.957299 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.957264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\"" Sep 16 20:04:05.958083 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.958060 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\"" Sep 16 20:04:05.958217 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.958181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\"" Sep 16 20:04:05.958570 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.958548 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:04:05.958670 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.958589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-cxp5k\"" Sep 16 20:04:05.958741 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.958594 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\"" Sep 16 20:04:05.968477 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:05.968452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:04:06.073804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.073804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.073971 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrmhz\" (UniqueName: \"kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.074041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.074335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.074114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175912 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175912 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.175912 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.175878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrmhz\" (UniqueName: \"kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.176061 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.176030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.176116 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.176077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.176168 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.176130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.176379 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.176362 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.176458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.176390 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.178268 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.178240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.178812 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.178793 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.189321 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.189252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrmhz\" (UniqueName: \"kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz\") pod \"router-with-refs-test-kserve-5d78697c9b-dqm9h\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.267873 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.267835 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:06.561259 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.561226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:04:06.562640 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:04:06.562612 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a428d0_ff12_4420_96f5_8df7cb49b134.slice/crio-f3b8870604edbefa52da8a58df40019ff000482d636b3574fd38329d144f96a0 WatchSource:0}: Error finding container f3b8870604edbefa52da8a58df40019ff000482d636b3574fd38329d144f96a0: Status 404 returned error can't find the container with id f3b8870604edbefa52da8a58df40019ff000482d636b3574fd38329d144f96a0 Sep 16 20:04:06.646476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.646435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerStarted","Data":"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a"} Sep 16 20:04:06.646622 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:06.646498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerStarted","Data":"f3b8870604edbefa52da8a58df40019ff000482d636b3574fd38329d144f96a0"} Sep 16 20:04:08.922198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:08.921343 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:04:09.665182 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:09.665147 2565 generic.go:356] "Generic (PLEG): container finished" podID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerID="448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a" exitCode=0 Sep 16 20:04:09.665452 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:09.665224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerDied","Data":"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a"} Sep 16 20:04:10.674801 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:10.674766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerStarted","Data":"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d"} Sep 16 20:04:10.699151 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:10.699092 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podStartSLOduration=3.387636312 podStartE2EDuration="5.699074105s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.311437793s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:10.697675234 +0000 UTC m=+3534.903630987" watchObservedRunningTime="2026-09-16 20:04:10.699074105 +0000 UTC m=+3534.905029862" Sep 16 20:04:11.268889 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:11.268851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:16.268049 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:16.268010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:04:16.269238 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:16.269211 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:04:23.949774 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:23.949728 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" containerID="cri-o://f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d" gracePeriod=45 Sep 16 20:04:24.479969 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.479916 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused" Sep 16 20:04:24.733118 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.733051 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:04:24.762970 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.762932 2565 generic.go:356] "Generic (PLEG): container finished" podID="ed09361f-0193-47dd-93ea-963367c14846" containerID="f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d" exitCode=0 Sep 16 20:04:24.763149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.762998 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" Sep 16 20:04:24.763149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.763019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerDied","Data":"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d"} Sep 16 20:04:24.763149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.763104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt" event={"ID":"ed09361f-0193-47dd-93ea-963367c14846","Type":"ContainerDied","Data":"87387c20dcbf1498e04023c975114b52ea9be320ac9f9a4c1dbd8dea7ce04129"} Sep 16 20:04:24.763149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.763128 2565 scope.go:117] "RemoveContainer" containerID="f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d" Sep 16 20:04:24.772821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.772799 2565 scope.go:117] "RemoveContainer" containerID="3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47" Sep 16 20:04:24.781538 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.781514 2565 scope.go:117] "RemoveContainer" containerID="f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d" Sep 16 20:04:24.781841 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:04:24.781813 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d\": container with ID starting with f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d not found: ID does not exist" containerID="f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d" Sep 16 20:04:24.781942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.781844 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d"} err="failed to get container status \"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d\": rpc error: code = NotFound desc = could not find container \"f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d\": container with ID starting with f84e5fddcf96a7becedd5f7babf8f61e85a6debf6c69ee2039a4462f5913133d not found: ID does not exist" Sep 16 20:04:24.781942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.781868 2565 scope.go:117] "RemoveContainer" containerID="3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47" Sep 16 20:04:24.782187 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:04:24.782162 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47\": container with ID starting with 3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47 not found: ID does not exist" containerID="3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47" Sep 16 20:04:24.782330 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.782203 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47"} err="failed to get container status \"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47\": rpc error: code = NotFound desc = could not find container \"3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47\": container with ID starting with 3a8872d51a98c59e530efca873d81d7c59431fa3be8c279893563e1190eb7a47 not found: ID does not exist" Sep 16 20:04:24.872021 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.871987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872045 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872065 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbfb5\" (UniqueName: \"kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872083 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872102 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872125 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872557 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872337 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs\") pod \"ed09361f-0193-47dd-93ea-963367c14846\" (UID: \"ed09361f-0193-47dd-93ea-963367c14846\") " Sep 16 20:04:24.872820 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:04:24.872820 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.872736 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home" (OuterVolumeSpecName: "home") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:04:24.873217 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.873183 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:04:24.874889 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.874867 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm" (OuterVolumeSpecName: "dshm") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:04:24.875371 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.875340 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5" (OuterVolumeSpecName: "kube-api-access-pbfb5") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "kube-api-access-pbfb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:04:24.875487 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.875386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:04:24.892674 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.892629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:04:24.897546 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.897514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed09361f-0193-47dd-93ea-963367c14846" (UID: "ed09361f-0193-47dd-93ea-963367c14846"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973798 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973829 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed09361f-0193-47dd-93ea-963367c14846-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973838 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973846 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed09361f-0193-47dd-93ea-963367c14846-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973853 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.973845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973860 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.974646 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973868 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbfb5\" (UniqueName: \"kubernetes.io/projected/ed09361f-0193-47dd-93ea-963367c14846-kube-api-access-pbfb5\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:24.974646 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:24.973877 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed09361f-0193-47dd-93ea-963367c14846-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:04:25.088316 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:25.088259 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:04:25.090695 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:25.090668 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-j6cnt"] Sep 16 20:04:26.255193 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:26.255146 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed09361f-0193-47dd-93ea-963367c14846" path="/var/lib/kubelet/pods/ed09361f-0193-47dd-93ea-963367c14846/volumes" Sep 16 20:04:26.269237 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:26.269193 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:04:31.643349 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.643306 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:04:31.643985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.643894 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" Sep 16 20:04:31.643985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.643920 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" Sep 16 20:04:31.643985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.643954 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="storage-initializer" Sep 16 20:04:31.643985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.643964 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="storage-initializer" Sep 16 20:04:31.644269 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.644073 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed09361f-0193-47dd-93ea-963367c14846" containerName="main" Sep 16 20:04:31.647425 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.647406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.650182 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.650159 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:04:31.650318 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.650159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\"" Sep 16 20:04:31.650318 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.650260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qqkdg\"" Sep 16 20:04:31.650458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.650352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Sep 16 20:04:31.650653 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.650638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Sep 16 20:04:31.651097 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.651075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Sep 16 20:04:31.665716 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.665684 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:04:31.735876 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.735829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxmtj\" (UniqueName: \"kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.735876 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.735874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.735935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.735966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.736013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.736036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.736068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.736144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.736102 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.836792 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.836754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.836999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.836832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxmtj\" (UniqueName: \"kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.836999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.836869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837136 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.836996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837136 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837136 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837330 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837330 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837330 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837515 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837576 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837634 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.837699 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.837630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.839681 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.839653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.839961 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.839937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.847368 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.847247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxmtj\" (UniqueName: \"kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj\") pod \"stop-feature-test-kserve-86c9894c98-dhr6t\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:31.958009 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:31.957919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:32.259971 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:32.259944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:04:32.260906 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:04:32.260883 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b209afe_dd32_4c57_964b_f5a7315f1d21.slice/crio-73189f8059c77f758929e34c2a56597d2977345384b0d6b8183fc418b2d1962f WatchSource:0}: Error finding container 73189f8059c77f758929e34c2a56597d2977345384b0d6b8183fc418b2d1962f: Status 404 returned error can't find the container with id 73189f8059c77f758929e34c2a56597d2977345384b0d6b8183fc418b2d1962f Sep 16 20:04:32.814658 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:32.814614 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerStarted","Data":"1c0e575c2059bcae3da061284f1b2640f598762253efdade73adfa1f81406fb2"} Sep 16 20:04:32.815100 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:32.814680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerStarted","Data":"73189f8059c77f758929e34c2a56597d2977345384b0d6b8183fc418b2d1962f"} Sep 16 20:04:34.828231 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:34.828196 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerID="1c0e575c2059bcae3da061284f1b2640f598762253efdade73adfa1f81406fb2" exitCode=0 Sep 16 20:04:34.828682 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:34.828293 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerDied","Data":"1c0e575c2059bcae3da061284f1b2640f598762253efdade73adfa1f81406fb2"} Sep 16 20:04:35.837356 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:35.837308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerStarted","Data":"3b4af13e6cb5bcbcd017517074eaebcf313205a621b6394b39ff303c6f22ca97"} Sep 16 20:04:35.862490 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:35.862396 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podStartSLOduration=2.443730255 podStartE2EDuration="4.862374093s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.418643838s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:35.861490693 +0000 UTC m=+3560.067446447" watchObservedRunningTime="2026-09-16 20:04:35.862374093 +0000 UTC m=+3560.068329848" Sep 16 20:04:35.958893 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:35.958855 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:36.268797 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:36.268752 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:04:41.958188 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:41.958140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:04:41.959821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:41.959790 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:04:46.268871 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:46.268806 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:04:51.959400 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:51.959348 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:04:56.268661 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:04:56.268608 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:01.959357 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:01.959301 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:06.268231 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:06.268171 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:11.958833 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:11.958730 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:16.268560 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:16.268509 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:21.959119 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:21.959058 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:26.268961 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:26.268911 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:31.958851 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:31.958801 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:36.269220 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:36.269177 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:41.959134 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:41.959084 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:46.268924 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:46.268875 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:05:51.959031 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:51.958980 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:05:56.269122 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:05:56.269076 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:06:01.959384 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:01.959333 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:06:06.268980 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:06.268939 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:06:11.958895 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:11.958852 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:06:16.268959 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:16.268904 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused" Sep 16 20:06:21.958720 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:21.958671 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:06:26.278968 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:26.278934 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:06:26.287206 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:26.287169 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:06:31.958879 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:31.958818 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:06:41.959332 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:41.959214 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused" Sep 16 20:06:51.968744 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:51.968707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:06:51.977109 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:51.977078 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:06:54.102911 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:06:54.102879 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:07:09.126610 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.126557 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" containerID="cri-o://3b4af13e6cb5bcbcd017517074eaebcf313205a621b6394b39ff303c6f22ca97" gracePeriod=45 Sep 16 20:07:09.821187 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.821152 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerID="3b4af13e6cb5bcbcd017517074eaebcf313205a621b6394b39ff303c6f22ca97" exitCode=0 Sep 16 20:07:09.821379 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.821230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerDied","Data":"3b4af13e6cb5bcbcd017517074eaebcf313205a621b6394b39ff303c6f22ca97"} Sep 16 20:07:09.879066 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.879039 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:07:09.967325 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967360 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967386 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967406 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967427 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967819 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxmtj\" (UniqueName: \"kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj\") pod \"0b209afe-dd32-4c57-964b-f5a7315f1d21\" (UID: \"0b209afe-dd32-4c57-964b-f5a7315f1d21\") " Sep 16 20:07:09.967819 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:07:09.967819 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.967801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:07:09.968132 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.968106 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home" (OuterVolumeSpecName: "home") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:07:09.970525 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.970492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:07:09.970632 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.970554 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm" (OuterVolumeSpecName: "dshm") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:07:09.970632 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.970558 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj" (OuterVolumeSpecName: "kube-api-access-nxmtj") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "kube-api-access-nxmtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:07:09.973933 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.973904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:07:09.981708 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:09.981668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b209afe-dd32-4c57-964b-f5a7315f1d21" (UID: "0b209afe-dd32-4c57-964b-f5a7315f1d21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:07:10.068174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068137 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068170 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b209afe-dd32-4c57-964b-f5a7315f1d21-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068178 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068186 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b209afe-dd32-4c57-964b-f5a7315f1d21-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068475 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068196 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068475 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068205 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxmtj\" (UniqueName: \"kubernetes.io/projected/0b209afe-dd32-4c57-964b-f5a7315f1d21-kube-api-access-nxmtj\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068475 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068214 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.068475 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.068223 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b209afe-dd32-4c57-964b-f5a7315f1d21-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:07:10.829218 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.829184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" Sep 16 20:07:10.829218 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.829200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t" event={"ID":"0b209afe-dd32-4c57-964b-f5a7315f1d21","Type":"ContainerDied","Data":"73189f8059c77f758929e34c2a56597d2977345384b0d6b8183fc418b2d1962f"} Sep 16 20:07:10.829873 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.829272 2565 scope.go:117] "RemoveContainer" containerID="3b4af13e6cb5bcbcd017517074eaebcf313205a621b6394b39ff303c6f22ca97" Sep 16 20:07:10.838721 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.838683 2565 scope.go:117] "RemoveContainer" containerID="1c0e575c2059bcae3da061284f1b2640f598762253efdade73adfa1f81406fb2" Sep 16 20:07:10.849867 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.849833 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:07:10.853084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:10.853059 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-86c9894c98-dhr6t"] Sep 16 20:07:12.253930 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:12.253893 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" path="/var/lib/kubelet/pods/0b209afe-dd32-4c57-964b-f5a7315f1d21/volumes" Sep 16 20:07:47.377985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.377951 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:07:47.378462 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.378414 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="storage-initializer" Sep 16 20:07:47.378462 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.378427 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="storage-initializer" Sep 16 20:07:47.378544 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.378471 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" Sep 16 20:07:47.378544 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.378477 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" Sep 16 20:07:47.378611 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.378552 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b209afe-dd32-4c57-964b-f5a7315f1d21" containerName="main" Sep 16 20:07:47.381560 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.381536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.384057 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.384034 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\"" Sep 16 20:07:47.384806 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.384786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:07:47.385025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.385005 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-mnpgk\"" Sep 16 20:07:47.385376 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.385348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\"" Sep 16 20:07:47.385519 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.385482 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\"" Sep 16 20:07:47.385588 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.385573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\"" Sep 16 20:07:47.398009 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.397979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:07:47.516634 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.516815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.516815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.516815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.516815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27n7j\" (UniqueName: \"kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.516815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.517022 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.517022 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.516886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617602 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617602 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27n7j\" (UniqueName: \"kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.617917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.617774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.618344 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.618173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.618344 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.618206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.618344 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.618262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.618477 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.618401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.618603 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.618585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.620182 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.620161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.620522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.620506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.626202 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.626180 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27n7j\" (UniqueName: \"kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j\") pod \"lora-single-adapter-test-kserve-d5855ddd-gn6hr\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:47.691637 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:47.691550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:48.037408 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:07:48.037364 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b2b282_b548_4a94_93b4_c5bb0daf32a9.slice/crio-f8959304310c7cfb61264f6a6d06577f61b623eebdf788db20d5357d6c7c7403 WatchSource:0}: Error finding container f8959304310c7cfb61264f6a6d06577f61b623eebdf788db20d5357d6c7c7403: Status 404 returned error can't find the container with id f8959304310c7cfb61264f6a6d06577f61b623eebdf788db20d5357d6c7c7403 Sep 16 20:07:48.044550 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:48.044521 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:07:49.050386 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:49.050353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerStarted","Data":"43574721f468c9aef0d7d22c26832c341a9a85f875fbd4fb4790cd0e4dcc5208"} Sep 16 20:07:49.050759 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:49.050408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerStarted","Data":"f8959304310c7cfb61264f6a6d06577f61b623eebdf788db20d5357d6c7c7403"} Sep 16 20:07:51.064483 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:51.064448 2565 generic.go:356] "Generic (PLEG): container finished" podID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerID="43574721f468c9aef0d7d22c26832c341a9a85f875fbd4fb4790cd0e4dcc5208" exitCode=0 Sep 16 20:07:51.064880 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:51.064519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerDied","Data":"43574721f468c9aef0d7d22c26832c341a9a85f875fbd4fb4790cd0e4dcc5208"} Sep 16 20:07:52.074391 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:52.074355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerStarted","Data":"e19969d27f97518fbd4877540f58625eaab5b16a0c7888f2da61efeb669e7e8f"} Sep 16 20:07:52.103958 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:52.103899 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podStartSLOduration=2.232808218 podStartE2EDuration="5.103882113s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.871073895s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:07:52.100374488 +0000 UTC m=+3756.306330240" watchObservedRunningTime="2026-09-16 20:07:52.103882113 +0000 UTC m=+3756.309837865" Sep 16 20:07:52.691853 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:52.691817 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:57.691855 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:57.691815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:07:57.693877 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:07:57.693846 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:07.692364 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:07.692245 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:17.692184 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:17.692137 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:27.692196 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:27.692144 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:35.327795 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:35.327759 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:08:37.692629 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:37.692581 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:47.692380 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:47.692327 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:08:50.358003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:50.357956 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" containerID="cri-o://9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d" gracePeriod=45 Sep 16 20:08:51.117698 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.117672 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:08:51.232828 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.232793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.232876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.232922 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.232962 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrmhz\" (UniqueName: \"kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233002 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233240 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233240 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233240 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home\") pod \"08a428d0-ff12-4420-96f5-8df7cb49b134\" (UID: \"08a428d0-ff12-4420-96f5-8df7cb49b134\") " Sep 16 20:08:51.233555 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233459 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:08:51.233982 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233946 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache" (OuterVolumeSpecName: "model-cache") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:08:51.234115 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.233979 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home" (OuterVolumeSpecName: "home") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:08:51.235590 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.235560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:08:51.235872 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.235847 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz" (OuterVolumeSpecName: "kube-api-access-mrmhz") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "kube-api-access-mrmhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:08:51.236054 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.236026 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm" (OuterVolumeSpecName: "dshm") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:08:51.240495 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.240464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:08:51.248432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.248408 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08a428d0-ff12-4420-96f5-8df7cb49b134" (UID: "08a428d0-ff12-4420-96f5-8df7cb49b134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:08:51.334039 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.333989 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a428d0-ff12-4420-96f5-8df7cb49b134-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334109 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334138 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrmhz\" (UniqueName: \"kubernetes.io/projected/08a428d0-ff12-4420-96f5-8df7cb49b134-kube-api-access-mrmhz\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334147 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08a428d0-ff12-4420-96f5-8df7cb49b134-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334156 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334172 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334184 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.334251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.334195 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08a428d0-ff12-4420-96f5-8df7cb49b134-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:08:51.447318 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.447258 2565 generic.go:356] "Generic (PLEG): container finished" podID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerID="9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d" exitCode=0 Sep 16 20:08:51.447811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.447330 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" Sep 16 20:08:51.447811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.447342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerDied","Data":"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d"} Sep 16 20:08:51.447811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.447415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h" event={"ID":"08a428d0-ff12-4420-96f5-8df7cb49b134","Type":"ContainerDied","Data":"f3b8870604edbefa52da8a58df40019ff000482d636b3574fd38329d144f96a0"} Sep 16 20:08:51.447811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.447449 2565 scope.go:117] "RemoveContainer" containerID="9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d" Sep 16 20:08:51.458351 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.458329 2565 scope.go:117] "RemoveContainer" containerID="448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a" Sep 16 20:08:51.467499 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.467476 2565 scope.go:117] "RemoveContainer" containerID="9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d" Sep 16 20:08:51.467780 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:08:51.467758 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d\": container with ID starting with 9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d not found: ID does not exist" containerID="9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d" Sep 16 20:08:51.467865 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.467788 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d"} err="failed to get container status \"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d\": rpc error: code = NotFound desc = could not find container \"9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d\": container with ID starting with 9bb387b4eb04a2ba77ff65d5013d7148d30c63e66ecff2046542e9217d6e721d not found: ID does not exist" Sep 16 20:08:51.467865 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.467806 2565 scope.go:117] "RemoveContainer" containerID="448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a" Sep 16 20:08:51.468051 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:08:51.468036 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a\": container with ID starting with 448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a not found: ID does not exist" containerID="448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a" Sep 16 20:08:51.468121 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.468052 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a"} err="failed to get container status \"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a\": rpc error: code = NotFound desc = could not find container \"448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a\": container with ID starting with 448b69da6aa8e221c36fcf7650ebc133ad8a9f0afa9f24618888b833a937c60a not found: ID does not exist" Sep 16 20:08:51.486974 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.486889 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:08:51.498808 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:51.498779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d78697c9b-dqm9h"] Sep 16 20:08:52.253532 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:52.253490 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" path="/var/lib/kubelet/pods/08a428d0-ff12-4420-96f5-8df7cb49b134/volumes" Sep 16 20:08:57.693005 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:08:57.692954 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:09:07.702175 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:07.702136 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:09:07.710189 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:07.710157 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:09:32.843856 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.843815 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:09:32.844266 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.844246 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" Sep 16 20:09:32.844266 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.844260 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" Sep 16 20:09:32.844381 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.844276 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="storage-initializer" Sep 16 20:09:32.844381 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.844303 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="storage-initializer" Sep 16 20:09:32.844381 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.844377 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a428d0-ff12-4420-96f5-8df7cb49b134" containerName="main" Sep 16 20:09:32.847523 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.847501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.849955 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.849932 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:09:32.850069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.849997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hhfxw\"" Sep 16 20:09:32.850953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.850928 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\"" Sep 16 20:09:32.851069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.850931 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\"" Sep 16 20:09:32.851139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.851051 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\"" Sep 16 20:09:32.852691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.852664 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\"" Sep 16 20:09:32.868939 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.868907 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:09:32.911985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.911947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.911985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.911981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912429 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsmc\" (UniqueName: \"kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:32.912429 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:32.912246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013496 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013496 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013767 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013767 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsmc\" (UniqueName: \"kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013767 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.013926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.014100 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.013949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.014100 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.014076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.014241 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.014218 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.014313 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.014250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.016182 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.016151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.016432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.016413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.022770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.022748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsmc\" (UniqueName: \"kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.028620 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.028600 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d5855ddd-gn6hr_69b2b282-b548-4a94-93b4-c5bb0daf32a9/storage-initializer/0.log" Sep 16 20:09:33.038119 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.038100 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d5855ddd-gn6hr_69b2b282-b548-4a94-93b4-c5bb0daf32a9/main/0.log" Sep 16 20:09:33.164019 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.163923 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:33.450424 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.450390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:09:33.451514 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:09:33.451486 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70677818_d87b_4930_a6cc_79809bb1ed9b.slice/crio-d53431484c01220e602d6c44feb1434e473b09c4ccf8bf7455dde60b73a08e9b WatchSource:0}: Error finding container d53431484c01220e602d6c44feb1434e473b09c4ccf8bf7455dde60b73a08e9b: Status 404 returned error can't find the container with id d53431484c01220e602d6c44feb1434e473b09c4ccf8bf7455dde60b73a08e9b Sep 16 20:09:33.718458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.718421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerStarted","Data":"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e"} Sep 16 20:09:33.718625 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:33.718487 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerStarted","Data":"d53431484c01220e602d6c44feb1434e473b09c4ccf8bf7455dde60b73a08e9b"} Sep 16 20:09:36.737925 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:36.737890 2565 generic.go:356] "Generic (PLEG): container finished" podID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerID="6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e" exitCode=0 Sep 16 20:09:36.738333 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:36.737958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerDied","Data":"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e"} Sep 16 20:09:37.750113 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:37.750015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerStarted","Data":"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68"} Sep 16 20:09:37.775514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:37.775452 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podStartSLOduration=3.399368306 podStartE2EDuration="5.775436716s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.37606841s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:37.773830439 +0000 UTC m=+3861.979786216" watchObservedRunningTime="2026-09-16 20:09:37.775436716 +0000 UTC m=+3861.981392469" Sep 16 20:09:43.164434 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:43.164391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:43.165034 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:43.164470 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:09:43.166405 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:43.166373 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:09:43.241964 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:43.241921 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:09:53.164607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:53.164553 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:09:58.265991 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.265950 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" containerID="cri-o://e19969d27f97518fbd4877540f58625eaab5b16a0c7888f2da61efeb669e7e8f" gracePeriod=45 Sep 16 20:09:58.703057 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.702957 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused" Sep 16 20:09:58.883783 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.883730 2565 generic.go:356] "Generic (PLEG): container finished" podID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerID="e19969d27f97518fbd4877540f58625eaab5b16a0c7888f2da61efeb669e7e8f" exitCode=0 Sep 16 20:09:58.883961 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.883861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerDied","Data":"e19969d27f97518fbd4877540f58625eaab5b16a0c7888f2da61efeb669e7e8f"} Sep 16 20:09:58.956651 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.956584 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:09:58.982497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27n7j\" (UniqueName: \"kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982546 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982653 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982704 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982731 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir\") pod \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\" (UID: \"69b2b282-b548-4a94-93b4-c5bb0daf32a9\") " Sep 16 20:09:58.982962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:09:58.982962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.982895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:09:58.983172 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.983123 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69b2b282-b548-4a94-93b4-c5bb0daf32a9-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:58.983172 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.983143 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:58.983523 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.983235 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home" (OuterVolumeSpecName: "home") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:09:58.985596 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.985548 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j" (OuterVolumeSpecName: "kube-api-access-27n7j") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "kube-api-access-27n7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:09:58.986061 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.985975 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:09:58.986635 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:58.986582 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm" (OuterVolumeSpecName: "dshm") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:09:59.004411 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.004366 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:09:59.017121 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.017087 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69b2b282-b548-4a94-93b4-c5bb0daf32a9" (UID: "69b2b282-b548-4a94-93b4-c5bb0daf32a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:09:59.084332 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084268 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27n7j\" (UniqueName: \"kubernetes.io/projected/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kube-api-access-27n7j\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.084332 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084328 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.084332 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084339 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.084562 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084349 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.084562 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084357 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.084562 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.084364 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b2b282-b548-4a94-93b4-c5bb0daf32a9-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:09:59.891811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.891775 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" Sep 16 20:09:59.891811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.891793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr" event={"ID":"69b2b282-b548-4a94-93b4-c5bb0daf32a9","Type":"ContainerDied","Data":"f8959304310c7cfb61264f6a6d06577f61b623eebdf788db20d5357d6c7c7403"} Sep 16 20:09:59.892270 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.891861 2565 scope.go:117] "RemoveContainer" containerID="e19969d27f97518fbd4877540f58625eaab5b16a0c7888f2da61efeb669e7e8f" Sep 16 20:09:59.901965 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.901944 2565 scope.go:117] "RemoveContainer" containerID="43574721f468c9aef0d7d22c26832c341a9a85f875fbd4fb4790cd0e4dcc5208" Sep 16 20:09:59.922598 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.922566 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:09:59.930759 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:09:59.930730 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d5855ddd-gn6hr"] Sep 16 20:10:00.254329 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:00.254264 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" path="/var/lib/kubelet/pods/69b2b282-b548-4a94-93b4-c5bb0daf32a9/volumes" Sep 16 20:10:03.164530 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:03.164482 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:07.960152 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960117 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:10:07.960613 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960593 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" Sep 16 20:10:07.960613 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960612 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" Sep 16 20:10:07.960709 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960668 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="storage-initializer" Sep 16 20:10:07.960709 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960675 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="storage-initializer" Sep 16 20:10:07.960773 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.960753 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b2b282-b548-4a94-93b4-c5bb0daf32a9" containerName="main" Sep 16 20:10:07.964055 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.964035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:07.967593 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.967563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\"" Sep 16 20:10:07.967725 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.967694 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:10:07.967857 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.967828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\"" Sep 16 20:10:07.967998 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.967873 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-qv2gj\"" Sep 16 20:10:07.967998 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.967969 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\"" Sep 16 20:10:07.968315 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.968295 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\"" Sep 16 20:10:07.979924 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:07.979888 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:10:08.077620 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077620 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077844 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077844 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4cn\" (UniqueName: \"kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077844 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.077953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.077937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4cn\" (UniqueName: \"kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179493 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179753 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179815 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179877 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179937 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179997 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.179997 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.179983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.180150 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.180128 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.180478 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.180453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.181962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.181937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.182717 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.182695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.188171 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.188144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4cn\" (UniqueName: \"kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn\") pod \"lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.275861 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.275815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:08.560463 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.560432 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:10:08.563760 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:10:08.563726 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580d3351_97a9_46c5_936b_c15ab5ee99f3.slice/crio-c6016f7483ec6ed3fb22a9133750fe03cbf1efafb67e26eadbcf0e4aeb4f5d9c WatchSource:0}: Error finding container c6016f7483ec6ed3fb22a9133750fe03cbf1efafb67e26eadbcf0e4aeb4f5d9c: Status 404 returned error can't find the container with id c6016f7483ec6ed3fb22a9133750fe03cbf1efafb67e26eadbcf0e4aeb4f5d9c Sep 16 20:10:08.951943 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.951853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerStarted","Data":"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5"} Sep 16 20:10:08.951943 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:08.951909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerStarted","Data":"c6016f7483ec6ed3fb22a9133750fe03cbf1efafb67e26eadbcf0e4aeb4f5d9c"} Sep 16 20:10:11.977340 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:11.977296 2565 generic.go:356] "Generic (PLEG): container finished" podID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerID="371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5" exitCode=0 Sep 16 20:10:11.977340 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:11.977321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerDied","Data":"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5"} Sep 16 20:10:12.988230 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:12.988189 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerStarted","Data":"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3"} Sep 16 20:10:13.011133 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:13.011064 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podStartSLOduration=2.75408268 podStartE2EDuration="6.011043386s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.256960706s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:10:13.009137444 +0000 UTC m=+3897.215093196" watchObservedRunningTime="2026-09-16 20:10:13.011043386 +0000 UTC m=+3897.216999140" Sep 16 20:10:13.165052 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:13.165004 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:13.276477 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:13.276392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:18.276443 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:18.276402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:10:18.278230 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:18.278193 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:10:23.165094 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:23.165047 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:28.277045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:28.276994 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:10:33.165148 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:33.165103 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:38.276948 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:38.276904 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:10:43.165274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:43.165224 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:48.277074 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:48.277023 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:10:53.165253 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:53.165198 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:10:58.276861 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:10:58.276814 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:11:03.164782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:03.164734 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:11:08.276958 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:08.276847 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:11:13.164369 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:13.164316 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:11:18.276776 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:18.276719 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:11:23.164718 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:23.164672 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:11:28.287416 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:28.287383 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:11:28.295412 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:28.295378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:11:33.164437 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:33.164383 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused" Sep 16 20:11:43.175743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:43.175707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:11:43.183818 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:43.183793 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:11:55.347657 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:11:55.347611 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:12:00.893880 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:00.893835 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl_580d3351-97a9-46c5-936b-c15ab5ee99f3/storage-initializer/0.log" Sep 16 20:12:00.905532 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:00.905488 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl_580d3351-97a9-46c5-936b-c15ab5ee99f3/main/0.log" Sep 16 20:12:01.969274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:01.969231 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:12:10.372866 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:10.372826 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" containerID="cri-o://f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68" gracePeriod=45 Sep 16 20:12:11.139524 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.139492 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:12:11.262144 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262109 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262366 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262427 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262376 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262487 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262435 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:11.262487 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262456 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262503 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:12:11.262663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsmc\" (UniqueName: \"kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262570 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home\") pod \"70677818-d87b-4930-a6cc-79809bb1ed9b\" (UID: \"70677818-d87b-4930-a6cc-79809bb1ed9b\") " Sep 16 20:12:11.262927 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262851 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70677818-d87b-4930-a6cc-79809bb1ed9b-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.262927 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.262870 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.263214 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.263171 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home" (OuterVolumeSpecName: "home") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:11.264812 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.264791 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm" (OuterVolumeSpecName: "dshm") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:11.265130 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.265109 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:12:11.265251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.265226 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc" (OuterVolumeSpecName: "kube-api-access-zhsmc") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "kube-api-access-zhsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:12:11.276197 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.276166 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:11.288178 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.288139 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70677818-d87b-4930-a6cc-79809bb1ed9b" (UID: "70677818-d87b-4930-a6cc-79809bb1ed9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:11.364025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.363982 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.364025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.364023 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.364025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.364035 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsmc\" (UniqueName: \"kubernetes.io/projected/70677818-d87b-4930-a6cc-79809bb1ed9b-kube-api-access-zhsmc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.364353 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.364044 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70677818-d87b-4930-a6cc-79809bb1ed9b-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.364353 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.364054 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.364353 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.364062 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70677818-d87b-4930-a6cc-79809bb1ed9b-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:11.739222 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.739182 2565 generic.go:356] "Generic (PLEG): container finished" podID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerID="f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68" exitCode=0 Sep 16 20:12:11.739737 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.739230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerDied","Data":"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68"} Sep 16 20:12:11.739737 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.739255 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" Sep 16 20:12:11.739737 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.739321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq" event={"ID":"70677818-d87b-4930-a6cc-79809bb1ed9b","Type":"ContainerDied","Data":"d53431484c01220e602d6c44feb1434e473b09c4ccf8bf7455dde60b73a08e9b"} Sep 16 20:12:11.739737 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.739352 2565 scope.go:117] "RemoveContainer" containerID="f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68" Sep 16 20:12:11.750336 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.750310 2565 scope.go:117] "RemoveContainer" containerID="6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e" Sep 16 20:12:11.759094 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.759070 2565 scope.go:117] "RemoveContainer" containerID="f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68" Sep 16 20:12:11.759431 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:12:11.759407 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68\": container with ID starting with f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68 not found: ID does not exist" containerID="f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68" Sep 16 20:12:11.759503 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.759440 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68"} err="failed to get container status \"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68\": rpc error: code = NotFound desc = could not find container \"f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68\": container with ID starting with f53df8ad638a8c5c076ddc0029b9c3ded388f158c688db6316d24a8521713e68 not found: ID does not exist" Sep 16 20:12:11.759503 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.759457 2565 scope.go:117] "RemoveContainer" containerID="6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e" Sep 16 20:12:11.759716 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:12:11.759696 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e\": container with ID starting with 6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e not found: ID does not exist" containerID="6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e" Sep 16 20:12:11.759763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.759721 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e"} err="failed to get container status \"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e\": rpc error: code = NotFound desc = could not find container \"6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e\": container with ID starting with 6d6db1aa0cef456543a4a0861e1beb89d9ee5e332b6f585521cd7dacf8e37b5e not found: ID does not exist" Sep 16 20:12:11.764255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.764227 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:12:11.769996 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:11.769970 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f74tgq"] Sep 16 20:12:12.254537 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:12.254504 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" path="/var/lib/kubelet/pods/70677818-d87b-4930-a6cc-79809bb1ed9b/volumes" Sep 16 20:12:16.992785 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:16.992744 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" containerID="cri-o://a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3" gracePeriod=45 Sep 16 20:12:17.288790 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.288670 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused" Sep 16 20:12:17.657920 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.657895 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:12:17.721898 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.721859 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.721898 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.721902 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.721926 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4cn\" (UniqueName: \"kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.721957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722306 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722306 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722424 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722311 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722424 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir\") pod \"580d3351-97a9-46c5-936b-c15ab5ee99f3\" (UID: \"580d3351-97a9-46c5-936b-c15ab5ee99f3\") " Sep 16 20:12:17.722424 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:12:17.722682 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home" (OuterVolumeSpecName: "home") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:17.722756 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722671 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/580d3351-97a9-46c5-936b-c15ab5ee99f3-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.722817 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.722793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:17.724440 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.724419 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:12:17.724628 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.724609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm" (OuterVolumeSpecName: "dshm") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:17.724943 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.724926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn" (OuterVolumeSpecName: "kube-api-access-5d4cn") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "kube-api-access-5d4cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:12:17.732822 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.732797 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:17.736411 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.736386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "580d3351-97a9-46c5-936b-c15ab5ee99f3" (UID: "580d3351-97a9-46c5-936b-c15ab5ee99f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:12:17.775627 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.775595 2565 generic.go:356] "Generic (PLEG): container finished" podID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerID="a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3" exitCode=0 Sep 16 20:12:17.775775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.775646 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" Sep 16 20:12:17.775775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.775681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerDied","Data":"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3"} Sep 16 20:12:17.775775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.775759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl" event={"ID":"580d3351-97a9-46c5-936b-c15ab5ee99f3","Type":"ContainerDied","Data":"c6016f7483ec6ed3fb22a9133750fe03cbf1efafb67e26eadbcf0e4aeb4f5d9c"} Sep 16 20:12:17.775929 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.775781 2565 scope.go:117] "RemoveContainer" containerID="a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3" Sep 16 20:12:17.785436 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.785417 2565 scope.go:117] "RemoveContainer" containerID="371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5" Sep 16 20:12:17.794267 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.794249 2565 scope.go:117] "RemoveContainer" containerID="a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3" Sep 16 20:12:17.794586 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:12:17.794565 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3\": container with ID starting with a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3 not found: ID does not exist" containerID="a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3" Sep 16 20:12:17.794651 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.794594 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3"} err="failed to get container status \"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3\": rpc error: code = NotFound desc = could not find container \"a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3\": container with ID starting with a713642ff0bb75bdf2f5d5d65f908ff3f3dc11400a8fc2d31dde69c5018f66a3 not found: ID does not exist" Sep 16 20:12:17.794651 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.794611 2565 scope.go:117] "RemoveContainer" containerID="371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5" Sep 16 20:12:17.794853 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:12:17.794835 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5\": container with ID starting with 371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5 not found: ID does not exist" containerID="371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5" Sep 16 20:12:17.794894 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.794861 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5"} err="failed to get container status \"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5\": rpc error: code = NotFound desc = could not find container \"371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5\": container with ID starting with 371f72b164e60cf6cc41b2e8980797384a8cc1524f99a0ec4025b438392e70e5 not found: ID does not exist" Sep 16 20:12:17.800438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.800416 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:12:17.807871 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.807846 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-644dfbc87c-sf8dl"] Sep 16 20:12:17.823522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823469 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823492 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823501 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823508 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580d3351-97a9-46c5-936b-c15ab5ee99f3-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823522 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823516 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4cn\" (UniqueName: \"kubernetes.io/projected/580d3351-97a9-46c5-936b-c15ab5ee99f3-kube-api-access-5d4cn\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823528 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:17.823726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:17.823537 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580d3351-97a9-46c5-936b-c15ab5ee99f3-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:18.253848 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:18.253810 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" path="/var/lib/kubelet/pods/580d3351-97a9-46c5-936b-c15ab5ee99f3/volumes" Sep 16 20:12:25.457412 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.457374 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78"] Sep 16 20:12:25.457958 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.457937 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.457961 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.457974 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.457982 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458000 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="storage-initializer" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458009 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="storage-initializer" Sep 16 20:12:25.458045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458038 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="storage-initializer" Sep 16 20:12:25.458394 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458047 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="storage-initializer" Sep 16 20:12:25.458394 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458151 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70677818-d87b-4930-a6cc-79809bb1ed9b" containerName="main" Sep 16 20:12:25.458394 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.458177 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="580d3351-97a9-46c5-936b-c15ab5ee99f3" containerName="main" Sep 16 20:12:25.463255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.463233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:25.466199 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.466166 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\"" Sep 16 20:12:25.467124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.467088 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\"" Sep 16 20:12:25.467232 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.467121 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-psgtf\"" Sep 16 20:12:25.467232 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.467129 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\"" Sep 16 20:12:25.470232 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.470208 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78"] Sep 16 20:12:25.594096 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.594057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrw4\" (UniqueName: \"kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:25.694899 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.694860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrw4\" (UniqueName: \"kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:25.703612 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:25.703578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrw4\" (UniqueName: \"kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.446995 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.446954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.455649 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.455619 2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.547761 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.547722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.547935 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.547864 2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.553661 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.553634 2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") DevicePath \"csi-9cbae9dd12fc726e1ab733ca780abba2602197105519ef0b78f7603d4dc80ea4\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.785335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.785244 2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a942aef81374026917978a93929cc4e0df7f09e2529c8d69c88150494aadae44/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:31.800415 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:31.800382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"e2e-pvc-lora-download-64k78\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:32.075018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:32.074931 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:32.362726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:32.362698 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78"] Sep 16 20:12:32.364273 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:12:32.364248 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78f1823b_378e_42ee_9229_e57f677e82f3.slice/crio-fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f WatchSource:0}: Error finding container fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f: Status 404 returned error can't find the container with id fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f Sep 16 20:12:32.862382 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:32.862343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" event={"ID":"78f1823b-378e-42ee-9229-e57f677e82f3","Type":"ContainerStarted","Data":"22c05fd231281c027e6f20aa28d9de6849bf701070e993a032869f9bec737169"} Sep 16 20:12:32.862844 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:32.862393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" event={"ID":"78f1823b-378e-42ee-9229-e57f677e82f3","Type":"ContainerStarted","Data":"fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f"} Sep 16 20:12:32.887268 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:32.887212 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" podStartSLOduration=10.887195409 podStartE2EDuration="10.887195409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:32.884037664 +0000 UTC m=+4037.089993416" watchObservedRunningTime="2026-09-16 20:12:32.887195409 +0000 UTC m=+4037.093151236" Sep 16 20:12:34.876170 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:34.876130 2565 generic.go:356] "Generic (PLEG): container finished" podID="78f1823b-378e-42ee-9229-e57f677e82f3" containerID="22c05fd231281c027e6f20aa28d9de6849bf701070e993a032869f9bec737169" exitCode=0 Sep 16 20:12:34.876591 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:34.876207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" event={"ID":"78f1823b-378e-42ee-9229-e57f677e82f3","Type":"ContainerDied","Data":"22c05fd231281c027e6f20aa28d9de6849bf701070e993a032869f9bec737169"} Sep 16 20:12:36.027462 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.027432 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:36.193785 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.193681 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfrw4\" (UniqueName: \"kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4\") pod \"78f1823b-378e-42ee-9229-e57f677e82f3\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " Sep 16 20:12:36.193952 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.193813 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"78f1823b-378e-42ee-9229-e57f677e82f3\" (UID: \"78f1823b-378e-42ee-9229-e57f677e82f3\") " Sep 16 20:12:36.196321 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.196269 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4" (OuterVolumeSpecName: "kube-api-access-pfrw4") pod "78f1823b-378e-42ee-9229-e57f677e82f3" (UID: "78f1823b-378e-42ee-9229-e57f677e82f3"). InnerVolumeSpecName "kube-api-access-pfrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:12:36.199127 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.199106 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26" (OuterVolumeSpecName: "model-storage") pod "78f1823b-378e-42ee-9229-e57f677e82f3" (UID: "78f1823b-378e-42ee-9229-e57f677e82f3"). InnerVolumeSpecName "pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76". PluginName "kubernetes.io/csi", VolumeGIDValue "" Sep 16 20:12:36.294702 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.294664 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrw4\" (UniqueName: \"kubernetes.io/projected/78f1823b-378e-42ee-9229-e57f677e82f3-kube-api-access-pfrw4\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:12:36.294854 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.294724 2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") on node \"ip-10-0-128-148.ec2.internal\" " Sep 16 20:12:36.890292 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.890259 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" Sep 16 20:12:36.890464 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.890317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78" event={"ID":"78f1823b-378e-42ee-9229-e57f677e82f3","Type":"ContainerDied","Data":"fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f"} Sep 16 20:12:36.890464 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:36.890357 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe43d2d2d962d075ab14fcc7dacc79a14557b013afad1ac0362b3e9f79a2128f" Sep 16 20:12:37.269963 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:37.269935 2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26") on node "ip-10-0-128-148.ec2.internal" Sep 16 20:12:37.303818 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:37.303740 2565 reconciler_common.go:299] "Volume detached for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"csi-9cbae9dd12fc726e1ab733ca780abba2602197105519ef0b78f7603d4dc80ea4\"" Sep 16 20:12:53.762523 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.762470 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:12:53.763122 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.763069 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f1823b-378e-42ee-9229-e57f677e82f3" containerName="storage-initializer" Sep 16 20:12:53.763122 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.763090 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f1823b-378e-42ee-9229-e57f677e82f3" containerName="storage-initializer" Sep 16 20:12:53.763252 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.763210 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f1823b-378e-42ee-9229-e57f677e82f3" containerName="storage-initializer" Sep 16 20:12:53.773498 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.773465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.776643 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.776616 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\"" Sep 16 20:12:53.776835 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.776809 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:12:53.776979 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.776744 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\"" Sep 16 20:12:53.777046 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.776697 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\"" Sep 16 20:12:53.777150 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.777130 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\"" Sep 16 20:12:53.777229 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.777153 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-psgtf\"" Sep 16 20:12:53.779552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.779490 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:12:53.852476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dx2f\" (UniqueName: \"kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852567 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.852723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.852663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953079 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dx2f\" (UniqueName: \"kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953722 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953956 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953892 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.953956 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.953908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.954226 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.954042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.954432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.954407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.955786 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.955767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.955977 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.955959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:12:53.962538 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:12:53.962518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dx2f\" (UniqueName: \"kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:01.927076 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:01.927037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:01.932928 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:01.932905 2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.028506 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.028466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.028691 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.028620 2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.032040 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.032022 2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") DevicePath \"csi-9cbae9dd12fc726e1ab733ca780abba2602197105519ef0b78f7603d4dc80ea4\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.126547 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.126513 2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a942aef81374026917978a93929cc4e0df7f09e2529c8d69c88150494aadae44/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.147916 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.147888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.188917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.188835 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:02.491406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:02.491374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:13:02.492422 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:13:02.492398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669fa274_7912_4338_af13_914265ed41e2.slice/crio-2af167e413bd761419ad375d4fe08c554e6a2dd7d3f73923fcaa464ce4238a5b WatchSource:0}: Error finding container 2af167e413bd761419ad375d4fe08c554e6a2dd7d3f73923fcaa464ce4238a5b: Status 404 returned error can't find the container with id 2af167e413bd761419ad375d4fe08c554e6a2dd7d3f73923fcaa464ce4238a5b Sep 16 20:13:03.047919 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:03.047877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerStarted","Data":"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18"} Sep 16 20:13:03.048362 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:03.047937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerStarted","Data":"2af167e413bd761419ad375d4fe08c554e6a2dd7d3f73923fcaa464ce4238a5b"} Sep 16 20:13:05.065363 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.061831 2565 generic.go:356] "Generic (PLEG): container finished" podID="669fa274-7912-4338-af13-914265ed41e2" containerID="54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18" exitCode=0 Sep 16 20:13:05.065363 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.061999 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerDied","Data":"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18"} Sep 16 20:13:05.227645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.227609 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:13:05.231684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.231639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.235059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235036 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:13:05.236089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235332 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zhrkw\"" Sep 16 20:13:05.236089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235209 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\"" Sep 16 20:13:05.236089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235603 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\"" Sep 16 20:13:05.236089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235080 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Sep 16 20:13:05.236089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.235926 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\"" Sep 16 20:13:05.248884 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.248853 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:13:05.257592 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257755 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257755 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257755 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257843 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.257942 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.258124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.257970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd9m\" (UniqueName: \"kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358469 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358469 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358469 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd9m\" (UniqueName: \"kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.358770 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.359059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.359059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.359059 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.358908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.359229 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.359115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.359457 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.359437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.360946 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.360920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.361371 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.361347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.367599 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.367571 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd9m\" (UniqueName: \"kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.546056 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.546012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:05.841796 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:05.841765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:13:05.843196 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:13:05.843161 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7facd59_ebf0_4334_8e19_d9cea57fd0d4.slice/crio-0b411b170c91bf2dfb02c6f89f022ecc22de71884ebc5e5c1d9683bb3f1cce35 WatchSource:0}: Error finding container 0b411b170c91bf2dfb02c6f89f022ecc22de71884ebc5e5c1d9683bb3f1cce35: Status 404 returned error can't find the container with id 0b411b170c91bf2dfb02c6f89f022ecc22de71884ebc5e5c1d9683bb3f1cce35 Sep 16 20:13:06.070866 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:06.070825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerStarted","Data":"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c"} Sep 16 20:13:06.071518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:06.070899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerStarted","Data":"0b411b170c91bf2dfb02c6f89f022ecc22de71884ebc5e5c1d9683bb3f1cce35"} Sep 16 20:13:06.073171 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:06.073144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerStarted","Data":"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628"} Sep 16 20:13:06.120780 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:06.120639 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podStartSLOduration=10.886645758 podStartE2EDuration="13.120617714s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.233971956s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:13:06.119871471 +0000 UTC m=+4070.325827250" watchObservedRunningTime="2026-09-16 20:13:06.120617714 +0000 UTC m=+4070.326573468" Sep 16 20:13:06.189519 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:06.189465 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:09.100216 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:09.100178 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerID="45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c" exitCode=0 Sep 16 20:13:09.100726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:09.100255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerDied","Data":"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c"} Sep 16 20:13:10.109250 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:10.109204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerStarted","Data":"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307"} Sep 16 20:13:10.133929 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:10.133869 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podStartSLOduration=2.631614638 podStartE2EDuration="5.133853189s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.502238551s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:13:10.130727243 +0000 UTC m=+4074.336682996" watchObservedRunningTime="2026-09-16 20:13:10.133853189 +0000 UTC m=+4074.339808943" Sep 16 20:13:12.189239 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:12.189195 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:13:12.191215 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:12.191179 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:13:15.546867 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:15.546821 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:15.547369 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:15.546902 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:13:15.549035 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:15.548998 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:13:22.190267 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:22.190215 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:13:25.547187 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:25.547135 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:13:32.190308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:32.190224 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:13:35.547631 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:35.547582 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:13:42.189918 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:42.189873 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:13:45.547518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:45.547458 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:13:52.189971 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:52.189921 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:13:55.546780 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:13:55.546724 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:02.189876 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:02.189820 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:14:05.547346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:05.547299 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:12.189580 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:12.189473 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused" Sep 16 20:14:15.547151 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:15.547094 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:22.199979 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:22.199946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:14:22.208350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:22.208322 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:14:25.546730 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:25.546685 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:35.546768 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:35.546711 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:45.547478 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:45.547432 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:14:55.547121 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:14:55.547072 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:15:05.546577 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:05.546531 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:15:12.015438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:12.015399 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd_669fa274-7912-4338-af13-914265ed41e2/storage-initializer/0.log" Sep 16 20:15:12.025993 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:12.025962 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd_669fa274-7912-4338-af13-914265ed41e2/main/0.log" Sep 16 20:15:13.159328 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:13.159244 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:15:15.546757 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:15.546710 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:15:25.546848 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:25.546792 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused" Sep 16 20:15:28.185005 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.184954 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" containerID="cri-o://623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628" gracePeriod=45 Sep 16 20:15:28.842398 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.842346 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:15:28.886492 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886407 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886492 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886476 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886699 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886524 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dx2f\" (UniqueName: \"kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886699 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886699 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886621 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886863 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.886922 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886871 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home" (OuterVolumeSpecName: "home") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:15:28.886922 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886891 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.887029 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886939 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.887029 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.886969 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm\") pod \"669fa274-7912-4338-af13-914265ed41e2\" (UID: \"669fa274-7912-4338-af13-914265ed41e2\") " Sep 16 20:15:28.887660 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.887298 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.887660 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.887300 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:15:28.887660 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.887580 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:15:28.889441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.889397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f" (OuterVolumeSpecName: "kube-api-access-8dx2f") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "kube-api-access-8dx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:15:28.889715 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.889674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:15:28.890062 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.890021 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm" (OuterVolumeSpecName: "dshm") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:15:28.893406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.893382 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76". PluginName "kubernetes.io/csi", VolumeGIDValue "" Sep 16 20:15:28.901069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.901047 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:15:28.913153 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.913127 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "669fa274-7912-4338-af13-914265ed41e2" (UID: "669fa274-7912-4338-af13-914265ed41e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:15:28.988210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988175 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669fa274-7912-4338-af13-914265ed41e2-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988210 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988203 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/669fa274-7912-4338-af13-914265ed41e2-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988231 2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") on node \"ip-10-0-128-148.ec2.internal\" " Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988242 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988250 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988258 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988265 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dx2f\" (UniqueName: \"kubernetes.io/projected/669fa274-7912-4338-af13-914265ed41e2-kube-api-access-8dx2f\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.988441 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.988273 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/669fa274-7912-4338-af13-914265ed41e2-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:28.998837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:28.998816 2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26") on node "ip-10-0-128-148.ec2.internal" Sep 16 20:15:29.029224 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.029188 2565 generic.go:356] "Generic (PLEG): container finished" podID="669fa274-7912-4338-af13-914265ed41e2" containerID="623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628" exitCode=0 Sep 16 20:15:29.029367 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.029237 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" Sep 16 20:15:29.029367 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.029265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerDied","Data":"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628"} Sep 16 20:15:29.029367 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.029348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd" event={"ID":"669fa274-7912-4338-af13-914265ed41e2","Type":"ContainerDied","Data":"2af167e413bd761419ad375d4fe08c554e6a2dd7d3f73923fcaa464ce4238a5b"} Sep 16 20:15:29.029479 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.029373 2565 scope.go:117] "RemoveContainer" containerID="623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628" Sep 16 20:15:29.039689 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.039667 2565 scope.go:117] "RemoveContainer" containerID="54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18" Sep 16 20:15:29.048262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.048244 2565 scope.go:117] "RemoveContainer" containerID="623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628" Sep 16 20:15:29.049033 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:15:29.049012 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628\": container with ID starting with 623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628 not found: ID does not exist" containerID="623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628" Sep 16 20:15:29.049099 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.049039 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628"} err="failed to get container status \"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628\": rpc error: code = NotFound desc = could not find container \"623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628\": container with ID starting with 623f468734432c801e3d2a353619b3a6264267977a64bab1395ef010c0a79628 not found: ID does not exist" Sep 16 20:15:29.049099 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.049055 2565 scope.go:117] "RemoveContainer" containerID="54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18" Sep 16 20:15:29.049360 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:15:29.049335 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18\": container with ID starting with 54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18 not found: ID does not exist" containerID="54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18" Sep 16 20:15:29.049413 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.049372 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18"} err="failed to get container status \"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18\": rpc error: code = NotFound desc = could not find container \"54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18\": container with ID starting with 54798958fd6f8e0ef44687db9fcb654e76721153cc1606302c618dd94b791f18 not found: ID does not exist" Sep 16 20:15:29.053558 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.053537 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:15:29.058501 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.058480 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-9c476bc4-8jppd"] Sep 16 20:15:29.089272 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:29.089251 2565 reconciler_common.go:299] "Volume detached for volume \"pvc-649ad9e3-6914-41d1-8d14-a2d6e1e6fb76\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cda25f988d6abc26\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"csi-9cbae9dd12fc726e1ab733ca780abba2602197105519ef0b78f7603d4dc80ea4\"" Sep 16 20:15:30.253621 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:30.253583 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669fa274-7912-4338-af13-914265ed41e2" path="/var/lib/kubelet/pods/669fa274-7912-4338-af13-914265ed41e2/volumes" Sep 16 20:15:35.557604 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:35.557568 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:15:35.565586 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:35.565554 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:15:36.713375 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.713328 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv"] Sep 16 20:15:36.714015 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.713992 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="storage-initializer" Sep 16 20:15:36.714118 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.714016 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="storage-initializer" Sep 16 20:15:36.714118 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.714031 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" Sep 16 20:15:36.714118 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.714039 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" Sep 16 20:15:36.714311 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.714132 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="669fa274-7912-4338-af13-914265ed41e2" containerName="main" Sep 16 20:15:36.717042 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.717021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:36.721077 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.721050 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\"" Sep 16 20:15:36.721226 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.721173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-7w9jk\"" Sep 16 20:15:36.721801 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.721783 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\"" Sep 16 20:15:36.721912 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.721869 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\"" Sep 16 20:15:36.774383 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.774345 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv"] Sep 16 20:15:36.857886 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.857841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnl7l\" (UniqueName: \"kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:36.959265 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.959226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnl7l\" (UniqueName: \"kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:36.968506 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:36.968473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnl7l\" (UniqueName: \"kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:38.295541 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:38.295502 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78"] Sep 16 20:15:38.301568 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:38.301540 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-64k78"] Sep 16 20:15:40.254678 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:40.254643 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f1823b-378e-42ee-9229-e57f677e82f3" path="/var/lib/kubelet/pods/78f1823b-378e-42ee-9229-e57f677e82f3/volumes" Sep 16 20:15:44.977508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:44.977472 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:15:45.540657 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.540618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.550733 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.550709 2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.641610 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.641571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.641811 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.641705 2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.646478 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.646453 2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") DevicePath \"csi-d2ba47c0f8edaab2b9e19f1a572b4339f1b58bc17e57df9e70242530a59b829f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.873783 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.873750 2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6df679f3e67f7e8e5d4880ce5e60d6f1eb465481774af1fc7c8c9625edf8b3e5/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:45.887392 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:45.887364 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"e2e-pvc-lora-download-2f9tv\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:46.028408 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:46.028372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:46.302147 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:46.302120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv"] Sep 16 20:15:46.304900 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:15:46.304870 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e518168_d9ad_4de1_82b0_20805df09073.slice/crio-46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b WatchSource:0}: Error finding container 46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b: Status 404 returned error can't find the container with id 46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b Sep 16 20:15:47.135413 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:47.135377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" event={"ID":"8e518168-d9ad-4de1-82b0-20805df09073","Type":"ContainerStarted","Data":"815d6f709db4c839fb5d9aa33d554870a368150ad8eb3d50f7dda1bb41cb142f"} Sep 16 20:15:47.135413 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:47.135420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" event={"ID":"8e518168-d9ad-4de1-82b0-20805df09073","Type":"ContainerStarted","Data":"46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b"} Sep 16 20:15:47.156240 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:47.156175 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" podStartSLOduration=14.156152744 podStartE2EDuration="14.156152744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:15:47.154475331 +0000 UTC m=+4231.360431079" watchObservedRunningTime="2026-09-16 20:15:47.156152744 +0000 UTC m=+4231.362108498" Sep 16 20:15:49.148893 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:49.148856 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e518168-d9ad-4de1-82b0-20805df09073" containerID="815d6f709db4c839fb5d9aa33d554870a368150ad8eb3d50f7dda1bb41cb142f" exitCode=0 Sep 16 20:15:49.149321 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:49.148930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" event={"ID":"8e518168-d9ad-4de1-82b0-20805df09073","Type":"ContainerDied","Data":"815d6f709db4c839fb5d9aa33d554870a368150ad8eb3d50f7dda1bb41cb142f"} Sep 16 20:15:50.301084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.301058 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:50.389354 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.389318 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnl7l\" (UniqueName: \"kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l\") pod \"8e518168-d9ad-4de1-82b0-20805df09073\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " Sep 16 20:15:50.389517 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.389444 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"8e518168-d9ad-4de1-82b0-20805df09073\" (UID: \"8e518168-d9ad-4de1-82b0-20805df09073\") " Sep 16 20:15:50.391782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.391749 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l" (OuterVolumeSpecName: "kube-api-access-tnl7l") pod "8e518168-d9ad-4de1-82b0-20805df09073" (UID: "8e518168-d9ad-4de1-82b0-20805df09073"). InnerVolumeSpecName "kube-api-access-tnl7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:15:50.392445 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.392424 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741" (OuterVolumeSpecName: "model-storage") pod "8e518168-d9ad-4de1-82b0-20805df09073" (UID: "8e518168-d9ad-4de1-82b0-20805df09073"). InnerVolumeSpecName "pvc-b3cac000-aca4-4493-b408-614469dd8829". PluginName "kubernetes.io/csi", VolumeGIDValue "" Sep 16 20:15:50.491005 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.490967 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnl7l\" (UniqueName: \"kubernetes.io/projected/8e518168-d9ad-4de1-82b0-20805df09073-kube-api-access-tnl7l\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:15:50.491181 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:50.491026 2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") on node \"ip-10-0-128-148.ec2.internal\" " Sep 16 20:15:51.161600 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:51.161570 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" Sep 16 20:15:51.161784 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:51.161615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv" event={"ID":"8e518168-d9ad-4de1-82b0-20805df09073","Type":"ContainerDied","Data":"46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b"} Sep 16 20:15:51.161784 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:51.161658 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46b0a88d4a6cbea25d29cb1981c6a02ded1d5fb4487d8c6159d1c44dc8b7b55b" Sep 16 20:15:51.464018 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:51.463991 2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b3cac000-aca4-4493-b408-614469dd8829" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741") on node "ip-10-0-128-148.ec2.internal" Sep 16 20:15:51.501152 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:15:51.501121 2565 reconciler_common.go:299] "Volume detached for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"csi-d2ba47c0f8edaab2b9e19f1a572b4339f1b58bc17e57df9e70242530a59b829f\"" Sep 16 20:16:00.006611 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.006573 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" containerID="cri-o://b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307" gracePeriod=45 Sep 16 20:16:00.688195 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.688167 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.786835 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.786888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.786925 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.786999 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.787040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.787063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.787093 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwd9m\" (UniqueName: \"kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.787850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.787175 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location\") pod \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\" (UID: \"f7facd59-ebf0-4334-8e19-d9cea57fd0d4\") " Sep 16 20:16:00.795504 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.795461 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:16:00.795762 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.795734 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:16:00.796076 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.796035 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home" (OuterVolumeSpecName: "home") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:16:00.797818 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.797789 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm" (OuterVolumeSpecName: "dshm") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:16:00.797926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.797896 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:16:00.798238 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.798220 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m" (OuterVolumeSpecName: "kube-api-access-qwd9m") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "kube-api-access-qwd9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:16:00.814580 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.814542 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:16:00.815861 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.815837 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7facd59-ebf0-4334-8e19-d9cea57fd0d4" (UID: "f7facd59-ebf0-4334-8e19-d9cea57fd0d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:16:00.889061 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889026 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889061 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889061 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889078 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889091 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889104 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889116 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889128 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:00.889255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:00.889141 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwd9m\" (UniqueName: \"kubernetes.io/projected/f7facd59-ebf0-4334-8e19-d9cea57fd0d4-kube-api-access-qwd9m\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:16:01.224321 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.224268 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerID="b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307" exitCode=0 Sep 16 20:16:01.224727 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.224344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerDied","Data":"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307"} Sep 16 20:16:01.224727 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.224360 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" Sep 16 20:16:01.224727 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.224439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r" event={"ID":"f7facd59-ebf0-4334-8e19-d9cea57fd0d4","Type":"ContainerDied","Data":"0b411b170c91bf2dfb02c6f89f022ecc22de71884ebc5e5c1d9683bb3f1cce35"} Sep 16 20:16:01.224727 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.224466 2565 scope.go:117] "RemoveContainer" containerID="b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307" Sep 16 20:16:01.235090 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.235066 2565 scope.go:117] "RemoveContainer" containerID="45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c" Sep 16 20:16:01.244229 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.244209 2565 scope.go:117] "RemoveContainer" containerID="b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307" Sep 16 20:16:01.244549 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:16:01.244530 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307\": container with ID starting with b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307 not found: ID does not exist" containerID="b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307" Sep 16 20:16:01.244618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.244559 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307"} err="failed to get container status \"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307\": rpc error: code = NotFound desc = could not find container \"b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307\": container with ID starting with b6fa990bc7b3866c2c8d2a79933a8cb332cdaded797f36a7d953efabeeca0307 not found: ID does not exist" Sep 16 20:16:01.244618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.244577 2565 scope.go:117] "RemoveContainer" containerID="45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c" Sep 16 20:16:01.244838 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:16:01.244824 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c\": container with ID starting with 45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c not found: ID does not exist" containerID="45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c" Sep 16 20:16:01.244883 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.244840 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c"} err="failed to get container status \"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c\": rpc error: code = NotFound desc = could not find container \"45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c\": container with ID starting with 45df3e2447e81228d16bbbb6cd83bdc5b6d303a341f9c6f9803a002e1bbc0a9c not found: ID does not exist" Sep 16 20:16:01.250684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.250658 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:16:01.254735 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:01.254715 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55bd7d675f-p4k5r"] Sep 16 20:16:02.253266 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:02.253233 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" path="/var/lib/kubelet/pods/f7facd59-ebf0-4334-8e19-d9cea57fd0d4/volumes" Sep 16 20:16:16.576882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.576846 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577322 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="storage-initializer" Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577336 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="storage-initializer" Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577347 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577353 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577364 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e518168-d9ad-4de1-82b0-20805df09073" containerName="storage-initializer" Sep 16 20:16:16.577387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577374 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e518168-d9ad-4de1-82b0-20805df09073" containerName="storage-initializer" Sep 16 20:16:16.577626 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577451 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e518168-d9ad-4de1-82b0-20805df09073" containerName="storage-initializer" Sep 16 20:16:16.577626 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.577462 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7facd59-ebf0-4334-8e19-d9cea57fd0d4" containerName="main" Sep 16 20:16:16.580570 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.580547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.583208 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.583183 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\"" Sep 16 20:16:16.583384 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.583216 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\"" Sep 16 20:16:16.583635 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.583611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-7w9jk\"" Sep 16 20:16:16.584086 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.584067 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\"" Sep 16 20:16:16.594889 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.594866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:16:16.636106 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.636066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.636106 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.636110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.636355 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.636143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.636355 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.636187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.636355 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.636223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.736743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.736702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.736928 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.736755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.736928 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.736813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737011 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.736948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737011 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.736997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737086 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.737067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cb6\" (UniqueName: \"kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737245 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.737226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737387 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.737365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.737447 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.737400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.739363 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.739338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.739693 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.739676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.837985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.837883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cb6\" (UniqueName: \"kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:16.847244 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:16.847217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cb6\" (UniqueName: \"kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.717523 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.717473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.728028 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.727985 2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.818222 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.818157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.818470 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.818346 2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.822006 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.821981 2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") DevicePath \"csi-d2ba47c0f8edaab2b9e19f1a572b4339f1b58bc17e57df9e70242530a59b829f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.913236 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.913205 2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6df679f3e67f7e8e5d4880ce5e60d6f1eb465481774af1fc7c8c9625edf8b3e5/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:26.934346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:26.934315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"lora-pvc-adapters-test-kserve-866bc56d7d-swtwb\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:27.091840 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:27.091738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:27.373958 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:27.373923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:16:27.374789 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:16:27.374744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572ecfe0_0e6e_4b92_88db_c82986191fdd.slice/crio-5b8dbe7d63b006800602289023b1c5ce48b18b54ffe3312a459d5f48270a8503 WatchSource:0}: Error finding container 5b8dbe7d63b006800602289023b1c5ce48b18b54ffe3312a459d5f48270a8503: Status 404 returned error can't find the container with id 5b8dbe7d63b006800602289023b1c5ce48b18b54ffe3312a459d5f48270a8503 Sep 16 20:16:28.386817 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:28.386780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" event={"ID":"572ecfe0-0e6e-4b92-88db-c82986191fdd","Type":"ContainerStarted","Data":"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e"} Sep 16 20:16:28.387231 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:28.386834 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" event={"ID":"572ecfe0-0e6e-4b92-88db-c82986191fdd","Type":"ContainerStarted","Data":"5b8dbe7d63b006800602289023b1c5ce48b18b54ffe3312a459d5f48270a8503"} Sep 16 20:16:29.092754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:29.092716 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:37.091977 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:37.091933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:16:37.093467 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:37.093429 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:16:47.093255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:47.093206 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:16:57.092934 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:16:57.092888 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:17:06.034352 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.034248 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podStartSLOduration=50.034229229 podStartE2EDuration="50.034229229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:16:28.415473155 +0000 UTC m=+4272.621428908" watchObservedRunningTime="2026-09-16 20:17:06.034229229 +0000 UTC m=+4310.240184985" Sep 16 20:17:06.037033 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.036997 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:17:06.040948 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.040927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.044667 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.044586 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\"" Sep 16 20:17:06.044809 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.044695 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:17:06.044886 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.044868 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\"" Sep 16 20:17:06.045308 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.045253 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-h5w44\"" Sep 16 20:17:06.045435 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.045312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Sep 16 20:17:06.045435 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.045357 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\"" Sep 16 20:17:06.058224 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.058197 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:17:06.142325 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142536 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142536 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5ht\" (UniqueName: \"kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142536 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142694 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142694 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142694 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.142846 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.142685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5ht\" (UniqueName: \"kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243505 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243853 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243919 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243989 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.243989 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.243929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.244323 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.244306 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.246310 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.246270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.246373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.246351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.260160 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.260131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5ht\" (UniqueName: \"kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht\") pod \"router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.356930 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.356834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:06.644723 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:06.644696 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:17:06.646082 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:17:06.646050 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5888714e_8b73_46fe_ae24_75986bc7d419.slice/crio-8dec3721adf768e440c9d322fbb9b8648c778ef2e2b33d423f518d1cd9886168 WatchSource:0}: Error finding container 8dec3721adf768e440c9d322fbb9b8648c778ef2e2b33d423f518d1cd9886168: Status 404 returned error can't find the container with id 8dec3721adf768e440c9d322fbb9b8648c778ef2e2b33d423f518d1cd9886168 Sep 16 20:17:07.093086 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:07.093040 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:17:07.635599 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:07.635561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerStarted","Data":"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6"} Sep 16 20:17:07.635822 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:07.635630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerStarted","Data":"8dec3721adf768e440c9d322fbb9b8648c778ef2e2b33d423f518d1cd9886168"} Sep 16 20:17:09.652084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:09.651986 2565 generic.go:356] "Generic (PLEG): container finished" podID="5888714e-8b73-46fe-ae24-75986bc7d419" containerID="3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6" exitCode=0 Sep 16 20:17:09.652084 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:09.652067 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerDied","Data":"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6"} Sep 16 20:17:10.661556 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:10.661523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerStarted","Data":"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92"} Sep 16 20:17:10.691953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:10.691890 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podStartSLOduration=2.173761633 podStartE2EDuration="4.691873111s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.518111478s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:10.690742423 +0000 UTC m=+4314.896698178" watchObservedRunningTime="2026-09-16 20:17:10.691873111 +0000 UTC m=+4314.897828863" Sep 16 20:17:16.357016 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:16.356973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:16.357528 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:16.357054 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:17:16.358760 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:16.358727 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:17:17.092863 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:17.092806 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:17:26.357903 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:26.357855 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:17:27.092327 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:27.092251 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:17:36.357407 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:36.357350 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:17:37.092224 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:37.092171 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:17:46.357720 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:46.357669 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:17:47.103129 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:47.103096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:17:47.111327 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:47.111300 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:17:56.357764 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:17:56.357718 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:06.357661 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:06.357609 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:14.789587 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:14.789553 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-866bc56d7d-swtwb_572ecfe0-0e6e-4b92-88db-c82986191fdd/main/0.log" Sep 16 20:18:15.998209 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:15.998172 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:18:16.358096 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:16.357995 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:26.357859 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:26.357813 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:31.022426 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:31.022361 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" containerID="cri-o://91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e" gracePeriod=45 Sep 16 20:18:31.104300 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:31.104244 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused" Sep 16 20:18:31.987115 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:31.987085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:18:32.009615 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62cb6\" (UniqueName: \"kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.009805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009638 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.009805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009660 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.009805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009723 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.009805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.009805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009782 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.010069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") pod \"572ecfe0-0e6e-4b92-88db-c82986191fdd\" (UID: \"572ecfe0-0e6e-4b92-88db-c82986191fdd\") " Sep 16 20:18:32.010069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.009959 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:18:32.010233 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.010213 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.010341 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.010317 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home" (OuterVolumeSpecName: "home") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:18:32.013051 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.013012 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6" (OuterVolumeSpecName: "kube-api-access-62cb6") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "kube-api-access-62cb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:18:32.013666 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.013630 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm" (OuterVolumeSpecName: "dshm") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:18:32.014548 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.014519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:18:32.022097 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.021874 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741" (OuterVolumeSpecName: "kserve-pvc-source") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "pvc-b3cac000-aca4-4493-b408-614469dd8829". PluginName "kubernetes.io/csi", VolumeGIDValue "" Sep 16 20:18:32.023542 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.023510 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "572ecfe0-0e6e-4b92-88db-c82986191fdd" (UID: "572ecfe0-0e6e-4b92-88db-c82986191fdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110828 2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") on node \"ip-10-0-128-148.ec2.internal\" " Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110856 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62cb6\" (UniqueName: \"kubernetes.io/projected/572ecfe0-0e6e-4b92-88db-c82986191fdd-kube-api-access-62cb6\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110865 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110875 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572ecfe0-0e6e-4b92-88db-c82986191fdd-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110883 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.110906 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.110890 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572ecfe0-0e6e-4b92-88db-c82986191fdd-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:18:32.142871 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.142846 2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b3cac000-aca4-4493-b408-614469dd8829" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741") on node "ip-10-0-128-148.ec2.internal" Sep 16 20:18:32.195141 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.195105 2565 generic.go:356] "Generic (PLEG): container finished" podID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerID="91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e" exitCode=0 Sep 16 20:18:32.195350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.195161 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" Sep 16 20:18:32.195350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.195192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" event={"ID":"572ecfe0-0e6e-4b92-88db-c82986191fdd","Type":"ContainerDied","Data":"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e"} Sep 16 20:18:32.195350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.195315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb" event={"ID":"572ecfe0-0e6e-4b92-88db-c82986191fdd","Type":"ContainerDied","Data":"5b8dbe7d63b006800602289023b1c5ce48b18b54ffe3312a459d5f48270a8503"} Sep 16 20:18:32.195501 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.195363 2565 scope.go:117] "RemoveContainer" containerID="91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e" Sep 16 20:18:32.206802 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.206775 2565 scope.go:117] "RemoveContainer" containerID="91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e" Sep 16 20:18:32.207115 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:18:32.207086 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e\": container with ID starting with 91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e not found: ID does not exist" containerID="91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e" Sep 16 20:18:32.207214 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.207121 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e"} err="failed to get container status \"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e\": rpc error: code = NotFound desc = could not find container \"91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e\": container with ID starting with 91d9014af9a7eb91495e9d3e9b04ebd29a63478832a4e4881bf95884107e7a4e not found: ID does not exist" Sep 16 20:18:32.211739 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.211715 2565 reconciler_common.go:299] "Volume detached for volume \"pvc-b3cac000-aca4-4493-b408-614469dd8829\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0faeae4a5b8cb1741\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"csi-d2ba47c0f8edaab2b9e19f1a572b4339f1b58bc17e57df9e70242530a59b829f\"" Sep 16 20:18:32.219874 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.219847 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:18:32.226107 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.226080 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-866bc56d7d-swtwb"] Sep 16 20:18:32.254839 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:32.254801 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" path="/var/lib/kubelet/pods/572ecfe0-0e6e-4b92-88db-c82986191fdd/volumes" Sep 16 20:18:33.461559 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:33.461526 2565 scope.go:117] "RemoveContainer" containerID="22c05fd231281c027e6f20aa28d9de6849bf701070e993a032869f9bec737169" Sep 16 20:18:36.358204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:36.358153 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:37.573262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.573174 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:18:37.573756 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.573618 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" Sep 16 20:18:37.573756 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.573630 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" Sep 16 20:18:37.573756 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.573717 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="572ecfe0-0e6e-4b92-88db-c82986191fdd" containerName="main" Sep 16 20:18:37.576150 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.576133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.578826 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.578801 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-rkmk2\"" Sep 16 20:18:37.593986 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.593953 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:18:37.660852 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.660815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661035 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.660919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661035 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.660999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.661045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.661071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.661095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5p7\" (UniqueName: \"kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.661121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.661139 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.661137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.761787 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.761742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.761787 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.761788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.761843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.761878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.761902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762050 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5p7\" (UniqueName: \"kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762250 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762250 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762630 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762630 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762630 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.762850 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.762726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.763359 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.763332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.764857 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.764822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.764971 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.764872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.774225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.774197 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5p7\" (UniqueName: \"kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7\") pod \"router-with-refs-pd-test-kserve-64d96dcdfb-2crv7\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:37.887069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:37.886965 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:38.207571 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:38.207545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:18:38.209389 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:18:38.209361 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce80ac6_087f_4de3_91e6_2994fb27d1ea.slice/crio-c517ff9fab4816ad31c91e4d68b81dad5831ddf19d0fcbd7548f0bbc1c9b1dbb WatchSource:0}: Error finding container c517ff9fab4816ad31c91e4d68b81dad5831ddf19d0fcbd7548f0bbc1c9b1dbb: Status 404 returned error can't find the container with id c517ff9fab4816ad31c91e4d68b81dad5831ddf19d0fcbd7548f0bbc1c9b1dbb Sep 16 20:18:38.236821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:38.236788 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerStarted","Data":"c517ff9fab4816ad31c91e4d68b81dad5831ddf19d0fcbd7548f0bbc1c9b1dbb"} Sep 16 20:18:39.244105 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:39.244063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerStarted","Data":"22d312ebc0e4ba5bc543ade3ef24f10cd5d6b7972a0a63baf63fc3bf82a94596"} Sep 16 20:18:39.244526 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:39.244225 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:40.258258 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:40.258213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerStarted","Data":"e1108eed0855005d09a1834b66174fad4c2d62e91efb85de60ce92522d1b2c68"} Sep 16 20:18:41.116123 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:41.116078 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv"] Sep 16 20:18:41.120789 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:41.120744 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-2f9tv"] Sep 16 20:18:42.254544 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:42.254508 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e518168-d9ad-4de1-82b0-20805df09073" path="/var/lib/kubelet/pods/8e518168-d9ad-4de1-82b0-20805df09073/volumes" Sep 16 20:18:42.271610 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:42.271569 2565 generic.go:356] "Generic (PLEG): container finished" podID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerID="e1108eed0855005d09a1834b66174fad4c2d62e91efb85de60ce92522d1b2c68" exitCode=0 Sep 16 20:18:42.271792 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:42.271655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerDied","Data":"e1108eed0855005d09a1834b66174fad4c2d62e91efb85de60ce92522d1b2c68"} Sep 16 20:18:43.282109 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:43.282068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerStarted","Data":"6b15d1c770bf3aad730571d094e1701ade0b1458ad51664e41a3948fa717fbda"} Sep 16 20:18:43.309597 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:43.309520 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podStartSLOduration=3.827334135 podStartE2EDuration="6.309498317s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.482164182s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:43.306093104 +0000 UTC m=+4407.512048858" watchObservedRunningTime="2026-09-16 20:18:43.309498317 +0000 UTC m=+4407.515454071" Sep 16 20:18:46.357782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:46.357734 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:47.887795 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:47.887729 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:47.888346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:47.887838 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:47.889517 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:47.889482 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:18:54.313173 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.313137 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:18:54.319551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.319523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.323149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323121 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:18:54.323305 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\"" Sep 16 20:18:54.323623 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323600 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\"" Sep 16 20:18:54.323814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323601 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\"" Sep 16 20:18:54.323814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323640 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\"" Sep 16 20:18:54.323814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.323713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-wsfb2\"" Sep 16 20:18:54.337550 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.337512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:18:54.431204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.431497 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.431424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532635 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.532837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.532872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.533058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533517 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.533492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533708 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.533493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533708 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.533513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.533868 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.533582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.536067 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.536037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.536555 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.536522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.545791 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.545761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv\") pod \"tls-verification-test-kserve-prefill-5c9754958-49b47\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.633051 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.633013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:18:54.926026 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:54.925996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:18:54.928433 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:18:54.928400 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582f724b_6cdf_42c1_a62a_ead7054e5e23.slice/crio-e8cf71882e2848c355af8ed2eecd221a0019087b290dce41a4e785112c6068b9 WatchSource:0}: Error finding container e8cf71882e2848c355af8ed2eecd221a0019087b290dce41a4e785112c6068b9: Status 404 returned error can't find the container with id e8cf71882e2848c355af8ed2eecd221a0019087b290dce41a4e785112c6068b9 Sep 16 20:18:55.368931 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:55.368886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerStarted","Data":"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc"} Sep 16 20:18:55.369486 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:55.368964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerStarted","Data":"e8cf71882e2848c355af8ed2eecd221a0019087b290dce41a4e785112c6068b9"} Sep 16 20:18:56.357973 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:56.357920 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused" Sep 16 20:18:57.888336 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:57.888199 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:18:57.906350 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:57.906306 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:18:58.389825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:58.389790 2565 generic.go:356] "Generic (PLEG): container finished" podID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerID="5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc" exitCode=0 Sep 16 20:18:58.390011 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:58.389866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerDied","Data":"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc"} Sep 16 20:18:59.399338 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:59.399272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerStarted","Data":"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc"} Sep 16 20:18:59.430470 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:18:59.430393 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podStartSLOduration=2.834260611 podStartE2EDuration="5.430363417s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.596102806s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:59.427633513 +0000 UTC m=+4423.633589334" watchObservedRunningTime="2026-09-16 20:18:59.430363417 +0000 UTC m=+4423.636319177" Sep 16 20:19:04.633533 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:04.633485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:19:04.634010 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:04.633571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:19:04.635650 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:04.635616 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:06.368165 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:06.368133 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:19:06.377408 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:06.377376 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:19:07.888069 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:07.887797 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:19:14.633750 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:14.633701 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:17.888235 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:17.888189 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:19:24.633821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:24.633774 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:27.888262 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:27.888199 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:19:34.634500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:34.634438 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:37.887512 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:37.887464 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:19:44.633630 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:44.633581 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:47.887482 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:47.887432 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:19:54.633609 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:54.633567 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:19:57.887581 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:19:57.887523 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:20:04.633882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:04.633832 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:20:07.887696 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:07.887582 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:20:14.634445 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:14.634390 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:20:17.887826 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:17.887754 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused" Sep 16 20:20:19.660955 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:19.660919 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:20:19.673334 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:19.673248 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:20:24.634029 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:24.633974 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:20:34.633719 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:34.633677 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:20:34.688810 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:34.688745 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" containerID="cri-o://30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92" gracePeriod=45 Sep 16 20:20:34.698268 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:34.698203 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" containerID="cri-o://6b15d1c770bf3aad730571d094e1701ade0b1458ad51664e41a3948fa717fbda" gracePeriod=45 Sep 16 20:20:35.952508 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:35.952478 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:20:36.065764 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.065667 2565 generic.go:356] "Generic (PLEG): container finished" podID="5888714e-8b73-46fe-ae24-75986bc7d419" containerID="30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92" exitCode=0 Sep 16 20:20:36.065764 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.065714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" Sep 16 20:20:36.065999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.065717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerDied","Data":"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92"} Sep 16 20:20:36.065999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.065849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h" event={"ID":"5888714e-8b73-46fe-ae24-75986bc7d419","Type":"ContainerDied","Data":"8dec3721adf768e440c9d322fbb9b8648c778ef2e2b33d423f518d1cd9886168"} Sep 16 20:20:36.065999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.065880 2565 scope.go:117] "RemoveContainer" containerID="30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92" Sep 16 20:20:36.075524 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.075504 2565 scope.go:117] "RemoveContainer" containerID="3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6" Sep 16 20:20:36.084239 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.084218 2565 scope.go:117] "RemoveContainer" containerID="30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92" Sep 16 20:20:36.084566 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:20:36.084549 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92\": container with ID starting with 30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92 not found: ID does not exist" containerID="30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92" Sep 16 20:20:36.084653 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.084574 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92"} err="failed to get container status \"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92\": rpc error: code = NotFound desc = could not find container \"30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92\": container with ID starting with 30495c7683dcdc5f380d105bda9d52efa6303e5c9f5d46591fb6b5278e32fc92 not found: ID does not exist" Sep 16 20:20:36.084653 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.084594 2565 scope.go:117] "RemoveContainer" containerID="3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6" Sep 16 20:20:36.084832 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:20:36.084815 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6\": container with ID starting with 3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6 not found: ID does not exist" containerID="3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6" Sep 16 20:20:36.084893 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.084837 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6"} err="failed to get container status \"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6\": rpc error: code = NotFound desc = could not find container \"3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6\": container with ID starting with 3674c5ab77e342170462b105c94a8ae494e10c1173908054a499752f710a26b6 not found: ID does not exist" Sep 16 20:20:36.087168 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087272 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087272 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087272 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087272 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087349 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087409 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5ht\" (UniqueName: \"kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht\") pod \"5888714e-8b73-46fe-ae24-75986bc7d419\" (UID: \"5888714e-8b73-46fe-ae24-75986bc7d419\") " Sep 16 20:20:36.087964 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.087872 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:20:36.088247 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.088222 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home" (OuterVolumeSpecName: "home") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:20:36.088730 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.088696 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache" (OuterVolumeSpecName: "model-cache") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:20:36.090076 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.090047 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:20:36.090181 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.090105 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm" (OuterVolumeSpecName: "dshm") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:20:36.090541 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.090513 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht" (OuterVolumeSpecName: "kube-api-access-nj5ht") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "kube-api-access-nj5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:20:36.094886 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.094862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:20:36.100133 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.100106 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5888714e-8b73-46fe-ae24-75986bc7d419" (UID: "5888714e-8b73-46fe-ae24-75986bc7d419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:20:36.189225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189186 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189219 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5888714e-8b73-46fe-ae24-75986bc7d419-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189228 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5ht\" (UniqueName: \"kubernetes.io/projected/5888714e-8b73-46fe-ae24-75986bc7d419-kube-api-access-nj5ht\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189225 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189236 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189245 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5888714e-8b73-46fe-ae24-75986bc7d419-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189253 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189261 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.189552 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.189269 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5888714e-8b73-46fe-ae24-75986bc7d419-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:20:36.385663 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.385563 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:20:36.390434 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:36.390397 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-67fc856cb5-qzj9h"] Sep 16 20:20:38.254151 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:38.254104 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" path="/var/lib/kubelet/pods/5888714e-8b73-46fe-ae24-75986bc7d419/volumes" Sep 16 20:20:44.633720 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:44.633674 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:20:54.633993 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:20:54.633939 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:21:04.633619 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:04.633560 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:21:14.634492 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:14.634426 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:21:19.674052 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:19.673999 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="llm-d-routing-sidecar" containerID="cri-o://22d312ebc0e4ba5bc543ade3ef24f10cd5d6b7972a0a63baf63fc3bf82a94596" gracePeriod=2 Sep 16 20:21:20.349949 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.349917 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-b5dd93e6_router-with-refs-pd-test-kserve-64d96dcdfb-2crv7_dce80ac6-087f-4de3-91e6-2994fb27d1ea/main/0.log" Sep 16 20:21:20.350794 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.350766 2565 generic.go:356] "Generic (PLEG): container finished" podID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerID="6b15d1c770bf3aad730571d094e1701ade0b1458ad51664e41a3948fa717fbda" exitCode=137 Sep 16 20:21:20.350794 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.350790 2565 generic.go:356] "Generic (PLEG): container finished" podID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerID="22d312ebc0e4ba5bc543ade3ef24f10cd5d6b7972a0a63baf63fc3bf82a94596" exitCode=0 Sep 16 20:21:20.350997 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.350841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerDied","Data":"6b15d1c770bf3aad730571d094e1701ade0b1458ad51664e41a3948fa717fbda"} Sep 16 20:21:20.350997 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.350908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerDied","Data":"22d312ebc0e4ba5bc543ade3ef24f10cd5d6b7972a0a63baf63fc3bf82a94596"} Sep 16 20:21:20.437509 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.437471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-b5dd93e6_router-with-refs-pd-test-kserve-64d96dcdfb-2crv7_dce80ac6-087f-4de3-91e6-2994fb27d1ea/main/0.log" Sep 16 20:21:20.439496 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.439473 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:21:20.618044 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.617954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618044 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618044 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618027 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618105 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618144 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618182 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618406 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618243 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd5p7\" (UniqueName: \"kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7\") pod \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\" (UID: \"dce80ac6-087f-4de3-91e6-2994fb27d1ea\") " Sep 16 20:21:20.618677 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618576 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:21:20.618677 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:21:20.618825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.618798 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home" (OuterVolumeSpecName: "home") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:21:20.620902 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.620874 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:21:20.621172 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.621153 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7" (OuterVolumeSpecName: "kube-api-access-kd5p7") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "kube-api-access-kd5p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:21:20.621255 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.621204 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm" (OuterVolumeSpecName: "dshm") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:21:20.623653 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.623629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:21:20.638058 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.638016 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dce80ac6-087f-4de3-91e6-2994fb27d1ea" (UID: "dce80ac6-087f-4de3-91e6-2994fb27d1ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:21:20.719880 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719834 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.719880 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719880 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719900 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719919 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719936 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dce80ac6-087f-4de3-91e6-2994fb27d1ea-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719952 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5p7\" (UniqueName: \"kubernetes.io/projected/dce80ac6-087f-4de3-91e6-2994fb27d1ea-kube-api-access-kd5p7\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719970 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:20.720551 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:20.719985 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce80ac6-087f-4de3-91e6-2994fb27d1ea-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:21:21.357999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.357972 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-b5dd93e6_router-with-refs-pd-test-kserve-64d96dcdfb-2crv7_dce80ac6-087f-4de3-91e6-2994fb27d1ea/main/0.log" Sep 16 20:21:21.358704 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.358676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" event={"ID":"dce80ac6-087f-4de3-91e6-2994fb27d1ea","Type":"ContainerDied","Data":"c517ff9fab4816ad31c91e4d68b81dad5831ddf19d0fcbd7548f0bbc1c9b1dbb"} Sep 16 20:21:21.358809 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.358703 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7" Sep 16 20:21:21.358809 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.358748 2565 scope.go:117] "RemoveContainer" containerID="6b15d1c770bf3aad730571d094e1701ade0b1458ad51664e41a3948fa717fbda" Sep 16 20:21:21.376249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.376225 2565 scope.go:117] "RemoveContainer" containerID="e1108eed0855005d09a1834b66174fad4c2d62e91efb85de60ce92522d1b2c68" Sep 16 20:21:21.387254 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.387230 2565 scope.go:117] "RemoveContainer" containerID="22d312ebc0e4ba5bc543ade3ef24f10cd5d6b7972a0a63baf63fc3bf82a94596" Sep 16 20:21:21.393944 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.393915 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:21:21.401478 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:21.401448 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-64d96dcdfb-2crv7"] Sep 16 20:21:22.253801 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:22.253755 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" path="/var/lib/kubelet/pods/dce80ac6-087f-4de3-91e6-2994fb27d1ea/volumes" Sep 16 20:21:24.644659 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:24.644626 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:21:24.652887 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:24.652857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:21:49.443356 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:21:49.443310 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:22:04.466809 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:04.466745 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" containerID="cri-o://643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc" gracePeriod=45 Sep 16 20:22:04.646337 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:04.646273 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused" Sep 16 20:22:05.222010 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.221983 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:22:05.313778 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.313778 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313763 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313787 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313807 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313823 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313855 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313884 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.313934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir\") pod \"582f724b-6cdf-42c1-a62a-ead7054e5e23\" (UID: \"582f724b-6cdf-42c1-a62a-ead7054e5e23\") " Sep 16 20:22:05.314397 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.314213 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache" (OuterVolumeSpecName: "model-cache") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.314457 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.314404 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:22:05.314790 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.314761 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home" (OuterVolumeSpecName: "home") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.316585 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.316557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:22:05.316585 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.316576 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv" (OuterVolumeSpecName: "kube-api-access-qncbv") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "kube-api-access-qncbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:22:05.316991 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.316969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm" (OuterVolumeSpecName: "dshm") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.329904 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.329876 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.337985 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.337960 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "582f724b-6cdf-42c1-a62a-ead7054e5e23" (UID: "582f724b-6cdf-42c1-a62a-ead7054e5e23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:22:05.415644 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415603 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415644 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415638 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582f724b-6cdf-42c1-a62a-ead7054e5e23-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415644 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415652 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/582f724b-6cdf-42c1-a62a-ead7054e5e23-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415663 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415675 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415699 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415710 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582f724b-6cdf-42c1-a62a-ead7054e5e23-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.415882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.415722 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncbv\" (UniqueName: \"kubernetes.io/projected/582f724b-6cdf-42c1-a62a-ead7054e5e23-kube-api-access-qncbv\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:22:05.623588 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.623503 2565 generic.go:356] "Generic (PLEG): container finished" podID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerID="643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc" exitCode=0 Sep 16 20:22:05.623588 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.623556 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" Sep 16 20:22:05.624046 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.623590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerDied","Data":"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc"} Sep 16 20:22:05.624046 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.623646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47" event={"ID":"582f724b-6cdf-42c1-a62a-ead7054e5e23","Type":"ContainerDied","Data":"e8cf71882e2848c355af8ed2eecd221a0019087b290dce41a4e785112c6068b9"} Sep 16 20:22:05.624046 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.623668 2565 scope.go:117] "RemoveContainer" containerID="643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc" Sep 16 20:22:05.633817 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.633799 2565 scope.go:117] "RemoveContainer" containerID="5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc" Sep 16 20:22:05.642153 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.642135 2565 scope.go:117] "RemoveContainer" containerID="643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc" Sep 16 20:22:05.642540 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:22:05.642519 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc\": container with ID starting with 643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc not found: ID does not exist" containerID="643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc" Sep 16 20:22:05.642625 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.642546 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc"} err="failed to get container status \"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc\": rpc error: code = NotFound desc = could not find container \"643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc\": container with ID starting with 643aef911f434121677fabbce8855f65afb6791be425e57c0d378fad0511f1dc not found: ID does not exist" Sep 16 20:22:05.642625 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.642562 2565 scope.go:117] "RemoveContainer" containerID="5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc" Sep 16 20:22:05.642845 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:22:05.642813 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc\": container with ID starting with 5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc not found: ID does not exist" containerID="5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc" Sep 16 20:22:05.642895 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.642849 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc"} err="failed to get container status \"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc\": rpc error: code = NotFound desc = could not find container \"5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc\": container with ID starting with 5fa903033fb48c9875eedcc44b8edb9185fef5e926b2287baa714012a02181cc not found: ID does not exist" Sep 16 20:22:05.647224 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.647200 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:22:05.654393 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:05.654370 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-5c9754958-49b47"] Sep 16 20:22:06.253274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:06.253239 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" path="/var/lib/kubelet/pods/582f724b-6cdf-42c1-a62a-ead7054e5e23/volumes" Sep 16 20:22:33.529466 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:33.529433 2565 scope.go:117] "RemoveContainer" containerID="815d6f709db4c839fb5d9aa33d554870a368150ad8eb3d50f7dda1bb41cb142f" Sep 16 20:22:39.939347 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939306 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:22:39.939914 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939895 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939918 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939930 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="llm-d-routing-sidecar" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939938 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="llm-d-routing-sidecar" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939949 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939956 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939974 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="storage-initializer" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939982 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="storage-initializer" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.939998 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="storage-initializer" Sep 16 20:22:39.940003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940006 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="storage-initializer" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940029 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940038 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940051 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="storage-initializer" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940060 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="storage-initializer" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940164 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5888714e-8b73-46fe-ae24-75986bc7d419" containerName="main" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940173 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="llm-d-routing-sidecar" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940185 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="582f724b-6cdf-42c1-a62a-ead7054e5e23" containerName="main" Sep 16 20:22:39.940500 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.940194 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce80ac6-087f-4de3-91e6-2994fb27d1ea" containerName="main" Sep 16 20:22:39.942568 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.942545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:39.945888 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.945862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\"" Sep 16 20:22:39.945888 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.945862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\"" Sep 16 20:22:39.948463 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.948434 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\"" Sep 16 20:22:39.948732 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.948711 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-5dlgk\"" Sep 16 20:22:39.948947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.948919 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\"" Sep 16 20:22:39.949470 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.949208 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:22:39.970706 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:39.970675 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:22:40.016548 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016548 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016765 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016910 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.016910 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.016814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhczj\" (UniqueName: \"kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhczj\" (UniqueName: \"kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118396 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118396 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118396 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118573 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118573 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118573 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118573 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118795 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118858 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118907 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.118907 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.118895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.119204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.119184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.121407 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.121380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.121594 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.121574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.132124 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.132096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhczj\" (UniqueName: \"kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.252854 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.252821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:40.536293 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.536247 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:22:40.537578 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:22:40.537554 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023b67bb_2ff8_4ee5_8cf3_ccb6bdc24ccf.slice/crio-908362c0881be88baedc23f9625c6389b27bc99c5e72ede3588395f9cda6d7cd WatchSource:0}: Error finding container 908362c0881be88baedc23f9625c6389b27bc99c5e72ede3588395f9cda6d7cd: Status 404 returned error can't find the container with id 908362c0881be88baedc23f9625c6389b27bc99c5e72ede3588395f9cda6d7cd Sep 16 20:22:40.821407 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.821324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerStarted","Data":"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955"} Sep 16 20:22:40.821407 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:40.821386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerStarted","Data":"908362c0881be88baedc23f9625c6389b27bc99c5e72ede3588395f9cda6d7cd"} Sep 16 20:22:43.840329 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:43.840267 2565 generic.go:356] "Generic (PLEG): container finished" podID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerID="e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955" exitCode=0 Sep 16 20:22:43.840722 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:43.840345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerDied","Data":"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955"} Sep 16 20:22:44.848466 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:44.848421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerStarted","Data":"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2"} Sep 16 20:22:44.876106 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:44.876036 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podStartSLOduration=3.613331754 podStartE2EDuration="5.876018235s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.262686481s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:22:44.873823762 +0000 UTC m=+4649.079779515" watchObservedRunningTime="2026-09-16 20:22:44.876018235 +0000 UTC m=+4649.081973988" Sep 16 20:22:45.253749 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:45.253700 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:50.253887 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:50.253850 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:22:50.255564 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:22:50.255529 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:00.254185 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:00.254127 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:10.253600 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.253501 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:10.589437 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.589347 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:23:10.593388 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.593363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.598243 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.598223 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\"" Sep 16 20:23:10.598375 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.598265 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\"" Sep 16 20:23:10.599509 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.599468 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 20:23:10.599637 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.599530 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\"" Sep 16 20:23:10.599807 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.599785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\"" Sep 16 20:23:10.599807 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.599788 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-45gvv\"" Sep 16 20:23:10.613910 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.613882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:23:10.711458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711684 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.711821 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.711741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99qg\" (UniqueName: \"kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.812805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.812805 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813077 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813077 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813077 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813077 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.812942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99qg\" (UniqueName: \"kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813422 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813422 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813422 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813646 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813646 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813483 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813799 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.813799 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.813791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.815672 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.815647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.815761 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.815703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.823833 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.823809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99qg\" (UniqueName: \"kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg\") pod \"prestop-hook-test-kserve-994b97cc4-87xbm\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:10.904655 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:10.904571 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:11.209249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:11.209221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:23:11.212275 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:23:11.212236 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda419a400_6987_4760_ae32_0869c9d3788b.slice/crio-f68db58829e9f83459b3def1bbac29afbafeeea4b31abce9530a33c38e9fbcb9 WatchSource:0}: Error finding container f68db58829e9f83459b3def1bbac29afbafeeea4b31abce9530a33c38e9fbcb9: Status 404 returned error can't find the container with id f68db58829e9f83459b3def1bbac29afbafeeea4b31abce9530a33c38e9fbcb9 Sep 16 20:23:12.012518 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:12.012465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerStarted","Data":"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e"} Sep 16 20:23:12.012995 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:12.012588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerStarted","Data":"f68db58829e9f83459b3def1bbac29afbafeeea4b31abce9530a33c38e9fbcb9"} Sep 16 20:23:14.027685 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:14.027592 2565 generic.go:356] "Generic (PLEG): container finished" podID="a419a400-6987-4760-ae32-0869c9d3788b" containerID="d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e" exitCode=0 Sep 16 20:23:14.027685 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:14.027651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerDied","Data":"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e"} Sep 16 20:23:15.037460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:15.037418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerStarted","Data":"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131"} Sep 16 20:23:15.065607 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:15.065547 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podStartSLOduration=2.591114556 podStartE2EDuration="5.065527693s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.474413137s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:23:15.063457918 +0000 UTC m=+4679.269413683" watchObservedRunningTime="2026-09-16 20:23:15.065527693 +0000 UTC m=+4679.271483446" Sep 16 20:23:15.905553 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:15.905515 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:20.254143 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:20.254086 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:20.905127 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:20.905087 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:23:20.907181 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:20.907150 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:23:30.253235 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:30.253186 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:30.906039 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:30.905987 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:23:40.253300 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:40.253230 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused" Sep 16 20:23:40.905332 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:40.905262 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:23:41.678618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:41.678577 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:23:50.906022 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:50.905975 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:23:56.704680 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:23:56.704607 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" containerID="cri-o://895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2" gracePeriod=45 Sep 16 20:24:00.905512 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:00.905459 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:24:10.905642 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:10.905594 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:24:20.905883 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:20.905832 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:24:30.905763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:30.905702 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:24:40.905701 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:40.905601 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:24:42.370198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.370168 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-fac1c4ae_llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7_023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf/main/0.log" Sep 16 20:24:42.373073 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.373049 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:24:42.519174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519076 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519174 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519140 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519182 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519241 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519275 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519354 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhczj\" (UniqueName: \"kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir\") pod \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\" (UID: \"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf\") " Sep 16 20:24:42.519763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519540 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:24:42.519763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:24:42.519763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519581 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home" (OuterVolumeSpecName: "home") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:24:42.519896 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519877 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.519962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519903 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.519962 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.519916 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.522211 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.522175 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:24:42.522544 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.522512 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm" (OuterVolumeSpecName: "dshm") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:24:42.522544 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.522520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj" (OuterVolumeSpecName: "kube-api-access-jhczj") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "kube-api-access-jhczj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:24:42.536149 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.536117 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:24:42.550274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.550244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" (UID: "023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:24:42.618158 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618125 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-fac1c4ae_llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7_023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf/main/0.log" Sep 16 20:24:42.618572 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618550 2565 generic.go:356] "Generic (PLEG): container finished" podID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerID="895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2" exitCode=137 Sep 16 20:24:42.618658 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerDied","Data":"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2"} Sep 16 20:24:42.618658 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618604 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" Sep 16 20:24:42.618658 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7" event={"ID":"023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf","Type":"ContainerDied","Data":"908362c0881be88baedc23f9625c6389b27bc99c5e72ede3588395f9cda6d7cd"} Sep 16 20:24:42.618658 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.618652 2565 scope.go:117] "RemoveContainer" containerID="895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2" Sep 16 20:24:42.620438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.620415 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhczj\" (UniqueName: \"kubernetes.io/projected/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kube-api-access-jhczj\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.620438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.620437 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.620617 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.620447 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.620617 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.620456 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.620617 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.620465 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:24:42.629347 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.629324 2565 scope.go:117] "RemoveContainer" containerID="e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955" Sep 16 20:24:42.639665 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.639647 2565 scope.go:117] "RemoveContainer" containerID="895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2" Sep 16 20:24:42.639965 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:24:42.639942 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2\": container with ID starting with 895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2 not found: ID does not exist" containerID="895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2" Sep 16 20:24:42.640045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.639978 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2"} err="failed to get container status \"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2\": rpc error: code = NotFound desc = could not find container \"895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2\": container with ID starting with 895c90baf6f29b4691c0fc3b11a13c98c74bbe43de090c5ee672a0bcbc3862a2 not found: ID does not exist" Sep 16 20:24:42.640045 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.640004 2565 scope.go:117] "RemoveContainer" containerID="e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955" Sep 16 20:24:42.640319 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:24:42.640293 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955\": container with ID starting with e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955 not found: ID does not exist" containerID="e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955" Sep 16 20:24:42.640768 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.640323 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955"} err="failed to get container status \"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955\": rpc error: code = NotFound desc = could not find container \"e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955\": container with ID starting with e5e5f198a74f8ebe0fe4056d08fed663d8426aa0a06618eeb645a343fb996955 not found: ID does not exist" Sep 16 20:24:42.647417 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.647368 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:24:42.651438 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:42.651413 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78d685c5d6qjfw7"] Sep 16 20:24:44.253614 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:44.253575 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" path="/var/lib/kubelet/pods/023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf/volumes" Sep 16 20:24:50.905517 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:24:50.905463 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:25:00.905185 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:00.905139 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:25:10.905677 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:10.905620 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:25:20.914682 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:20.914651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:25:20.922292 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:20.922248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:25:25.734398 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.734361 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:25:25.769113 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769069 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:25:25.769527 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769513 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="storage-initializer" Sep 16 20:25:25.769586 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769530 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="storage-initializer" Sep 16 20:25:25.769586 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769540 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" Sep 16 20:25:25.769586 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769545 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" Sep 16 20:25:25.769682 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.769617 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="023b67bb-2ff8-4ee5-8cf3-ccb6bdc24ccf" containerName="main" Sep 16 20:25:25.774220 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.774193 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.788730 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.788702 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:25:25.942899 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.942865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943070 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.942916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943070 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.942982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943070 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.943018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943070 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.943039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943070 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.943058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943247 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.943117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:25.943247 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:25.943149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfz2\" (UniqueName: \"kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044521 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044521 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044521 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044521 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044521 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfz2\" (UniqueName: \"kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044782 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.044986 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.044964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.045039 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.045016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.045354 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.045334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.047108 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.047076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.047390 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.047372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.053376 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.053351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfz2\" (UniqueName: \"kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2\") pod \"prestop-hook-test-kserve-994b97cc4-8nphm\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.085233 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.085191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:26.368120 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.368094 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:25:26.368775 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:25:26.368744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fab4fb_0778_4034_b89f_5c8ced6c665d.slice/crio-7c62fdc40e0ce70ce299c0b5aa4544831501a7b257ef28758682994024c4ed90 WatchSource:0}: Error finding container 7c62fdc40e0ce70ce299c0b5aa4544831501a7b257ef28758682994024c4ed90: Status 404 returned error can't find the container with id 7c62fdc40e0ce70ce299c0b5aa4544831501a7b257ef28758682994024c4ed90 Sep 16 20:25:26.890343 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.890305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerStarted","Data":"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3"} Sep 16 20:25:26.890868 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:26.890368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerStarted","Data":"7c62fdc40e0ce70ce299c0b5aa4544831501a7b257ef28758682994024c4ed90"} Sep 16 20:25:28.904109 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:28.904075 2565 generic.go:356] "Generic (PLEG): container finished" podID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerID="63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3" exitCode=0 Sep 16 20:25:28.904584 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:28.904153 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerDied","Data":"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3"} Sep 16 20:25:29.914543 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:29.914503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerStarted","Data":"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859"} Sep 16 20:25:29.939699 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:29.939643 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" podStartSLOduration=2.618379131 podStartE2EDuration="4.93962479s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.321245659s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:25:29.936352872 +0000 UTC m=+4814.142308625" watchObservedRunningTime="2026-09-16 20:25:29.93962479 +0000 UTC m=+4814.145580606" Sep 16 20:25:30.085483 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:30.085444 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:36.085585 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:36.085540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:25:36.087429 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:36.087398 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused" Sep 16 20:25:40.757696 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:40.757635 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" containerID="cri-o://5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131" gracePeriod=45 Sep 16 20:25:40.916073 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:40.916033 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused" Sep 16 20:25:41.527784 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.527718 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:25:41.582726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582694 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.582918 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.582918 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582815 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.582918 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582866 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.582918 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582917 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.583164 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.582959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v99qg\" (UniqueName: \"kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.583164 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.583000 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.583164 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.583040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir\") pod \"a419a400-6987-4760-ae32-0869c9d3788b\" (UID: \"a419a400-6987-4760-ae32-0869c9d3788b\") " Sep 16 20:25:41.583700 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.583514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home" (OuterVolumeSpecName: "home") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:25:41.584089 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.583900 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:25:41.584224 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.584191 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:25:41.589837 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.589794 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm" (OuterVolumeSpecName: "dshm") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:25:41.589959 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.589838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg" (OuterVolumeSpecName: "kube-api-access-v99qg") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "kube-api-access-v99qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:25:41.591639 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.591607 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:25:41.603267 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.603230 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:25:41.613843 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.613807 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a419a400-6987-4760-ae32-0869c9d3788b" (UID: "a419a400-6987-4760-ae32-0869c9d3788b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:25:41.684274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684234 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684263 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684277 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a419a400-6987-4760-ae32-0869c9d3788b-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684274 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684303 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684316 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419a400-6987-4760-ae32-0869c9d3788b-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684326 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v99qg\" (UniqueName: \"kubernetes.io/projected/a419a400-6987-4760-ae32-0869c9d3788b-kube-api-access-v99qg\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684335 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.684645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.684343 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419a400-6987-4760-ae32-0869c9d3788b-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:25:41.991580 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.991545 2565 generic.go:356] "Generic (PLEG): container finished" podID="a419a400-6987-4760-ae32-0869c9d3788b" containerID="5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131" exitCode=0 Sep 16 20:25:41.991954 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.991634 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" Sep 16 20:25:41.991954 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.991632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerDied","Data":"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131"} Sep 16 20:25:41.991954 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.991789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm" event={"ID":"a419a400-6987-4760-ae32-0869c9d3788b","Type":"ContainerDied","Data":"f68db58829e9f83459b3def1bbac29afbafeeea4b31abce9530a33c38e9fbcb9"} Sep 16 20:25:41.991954 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:41.991814 2565 scope.go:117] "RemoveContainer" containerID="5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131" Sep 16 20:25:42.001346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.001323 2565 scope.go:117] "RemoveContainer" containerID="d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e" Sep 16 20:25:42.011923 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.011898 2565 scope.go:117] "RemoveContainer" containerID="5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131" Sep 16 20:25:42.012233 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:25:42.012212 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131\": container with ID starting with 5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131 not found: ID does not exist" containerID="5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131" Sep 16 20:25:42.012458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.012250 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131"} err="failed to get container status \"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131\": rpc error: code = NotFound desc = could not find container \"5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131\": container with ID starting with 5a9feee28063c0ae5d7c0a396cef5e68ad71ce589c79821192a238ee647d4131 not found: ID does not exist" Sep 16 20:25:42.012458 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.012268 2565 scope.go:117] "RemoveContainer" containerID="d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e" Sep 16 20:25:42.012590 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:25:42.012572 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e\": container with ID starting with d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e not found: ID does not exist" containerID="d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e" Sep 16 20:25:42.012639 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.012592 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e"} err="failed to get container status \"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e\": rpc error: code = NotFound desc = could not find container \"d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e\": container with ID starting with d03018d13b7cdb9c42226874f2fb5b3f5d4748736667f936b7e959745792867e not found: ID does not exist" Sep 16 20:25:42.019799 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.019775 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:25:42.022943 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.022921 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-87xbm"] Sep 16 20:25:42.256138 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:42.256046 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a419a400-6987-4760-ae32-0869c9d3788b" path="/var/lib/kubelet/pods/a419a400-6987-4760-ae32-0869c9d3788b/volumes" Sep 16 20:25:46.085917 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:46.085871 2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused" Sep 16 20:25:47.497461 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:47.497427 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:25:49.244648 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.244605 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:25:49.245230 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.245208 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="storage-initializer" Sep 16 20:25:49.245346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.245233 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="storage-initializer" Sep 16 20:25:49.245346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.245251 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" Sep 16 20:25:49.245346 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.245258 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" Sep 16 20:25:49.245517 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.245403 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a419a400-6987-4760-ae32-0869c9d3788b" containerName="main" Sep 16 20:25:49.250345 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.250313 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.252882 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.252863 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x7n4c\"" Sep 16 20:25:49.253957 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.253929 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\"" Sep 16 20:25:49.254061 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.254032 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Sep 16 20:25:49.254198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.254133 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\"" Sep 16 20:25:49.257372 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.257348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:25:49.355754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.355947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5n66\" (UniqueName: \"kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.355947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.355947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.355947 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.356170 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.355947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.456763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.456725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5n66\" (UniqueName: \"kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.456763 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.456765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.456798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.456818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.456833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457025 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.457006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457256 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.457232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.457350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.457442 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.457376 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.459539 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.459515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.459745 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.459728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.465156 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.465133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5n66\" (UniqueName: \"kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-dlstb\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.563137 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.563043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:49.847833 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:49.847801 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:25:49.849485 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:25:49.849457 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cbc780_a5be_4823_8fce_afe003c2fc77.slice/crio-69ae3d82d7f4d77590601cbe0efd831c71042ae277c17cfb2fd20914c47a2597 WatchSource:0}: Error finding container 69ae3d82d7f4d77590601cbe0efd831c71042ae277c17cfb2fd20914c47a2597: Status 404 returned error can't find the container with id 69ae3d82d7f4d77590601cbe0efd831c71042ae277c17cfb2fd20914c47a2597 Sep 16 20:25:50.042044 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:50.042004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" event={"ID":"f8cbc780-a5be-4823-8fce-afe003c2fc77","Type":"ContainerStarted","Data":"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee"} Sep 16 20:25:50.042236 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:50.042066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" event={"ID":"f8cbc780-a5be-4823-8fce-afe003c2fc77","Type":"ContainerStarted","Data":"69ae3d82d7f4d77590601cbe0efd831c71042ae277c17cfb2fd20914c47a2597"} Sep 16 20:25:50.059618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:50.059562 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" podStartSLOduration=1.059545733 podStartE2EDuration="1.059545733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:25:50.059254702 +0000 UTC m=+4834.265210458" watchObservedRunningTime="2026-09-16 20:25:50.059545733 +0000 UTC m=+4834.265501486" Sep 16 20:25:50.563641 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:50.563603 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:59.563606 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:59.563553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:25:59.576762 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:25:59.576732 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:26:00.120735 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:00.120702 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:26:02.523098 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:02.523048 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" containerID="cri-o://c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859" gracePeriod=45 Sep 16 20:26:02.561099 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:02.561060 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:26:02.572098 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:26:02.572054 2565 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 16 20:26:02.572098 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 20:26:02.572098 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 20:26:02.572098 ip-10-0-128-148 kubenswrapper[2565]: > containerID="f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" cmd=["/bin/sleep","15"] Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:02.572118 2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" message="" err=< Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: > Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:26:02.572135 2565 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: , stdout: , stderr: , exit code -1 Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerName="main" containerID="cri-o://f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" Sep 16 20:26:02.572454 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:02.572241 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerName="main" containerID="cri-o://f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" gracePeriod=60 Sep 16 20:26:02.950100 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:02.950074 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:26:03.086780 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.086966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086802 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5n66\" (UniqueName: \"kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.086966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086849 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.086966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086885 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.086966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086923 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.086966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086942 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home" (OuterVolumeSpecName: "home") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:03.087227 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.086971 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs\") pod \"f8cbc780-a5be-4823-8fce-afe003c2fc77\" (UID: \"f8cbc780-a5be-4823-8fce-afe003c2fc77\") " Sep 16 20:26:03.087227 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.087112 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:03.087227 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.087132 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:03.087365 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.087271 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:03.087365 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.087314 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:03.087365 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.087323 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:03.089459 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.089422 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:26:03.089603 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.089583 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm" (OuterVolumeSpecName: "dshm") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:03.089688 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.089603 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66" (OuterVolumeSpecName: "kube-api-access-p5n66") pod "f8cbc780-a5be-4823-8fce-afe003c2fc77" (UID: "f8cbc780-a5be-4823-8fce-afe003c2fc77"). InnerVolumeSpecName "kube-api-access-p5n66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:26:03.131343 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.131307 2565 generic.go:356] "Generic (PLEG): container finished" podID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerID="f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" exitCode=0 Sep 16 20:26:03.131502 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.131371 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" Sep 16 20:26:03.131502 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.131371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" event={"ID":"f8cbc780-a5be-4823-8fce-afe003c2fc77","Type":"ContainerDied","Data":"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee"} Sep 16 20:26:03.131502 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.131491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb" event={"ID":"f8cbc780-a5be-4823-8fce-afe003c2fc77","Type":"ContainerDied","Data":"69ae3d82d7f4d77590601cbe0efd831c71042ae277c17cfb2fd20914c47a2597"} Sep 16 20:26:03.131612 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.131514 2565 scope.go:117] "RemoveContainer" containerID="f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" Sep 16 20:26:03.141492 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.141466 2565 scope.go:117] "RemoveContainer" containerID="f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" Sep 16 20:26:03.141856 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:26:03.141826 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee\": container with ID starting with f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee not found: ID does not exist" containerID="f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee" Sep 16 20:26:03.141945 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.141865 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee"} err="failed to get container status \"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee\": rpc error: code = NotFound desc = could not find container \"f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee\": container with ID starting with f8798d5bed3c83cd8d3d724bea34f54a395b9fd7e04e29efc4447deaa79e74ee not found: ID does not exist" Sep 16 20:26:03.156964 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.156917 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:26:03.161130 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.161102 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-dlstb"] Sep 16 20:26:03.188432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.188394 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5n66\" (UniqueName: \"kubernetes.io/projected/f8cbc780-a5be-4823-8fce-afe003c2fc77-kube-api-access-p5n66\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:03.188432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.188420 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8cbc780-a5be-4823-8fce-afe003c2fc77-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:03.188432 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:03.188431 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cbc780-a5be-4823-8fce-afe003c2fc77-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:04.255498 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:04.255458 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" path="/var/lib/kubelet/pods/f8cbc780-a5be-4823-8fce-afe003c2fc77/volumes" Sep 16 20:26:48.175127 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.175101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-994b97cc4-8nphm_39fab4fb-0778-4034-b89f-5c8ced6c665d/main/0.log" Sep 16 20:26:48.177775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.177752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:26:48.320476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320421 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320476 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320527 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rfz2\" (UniqueName: \"kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320627 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320656 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.320775 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320682 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache\") pod \"39fab4fb-0778-4034-b89f-5c8ced6c665d\" (UID: \"39fab4fb-0778-4034-b89f-5c8ced6c665d\") " Sep 16 20:26:48.321032 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320876 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home" (OuterVolumeSpecName: "home") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:48.321032 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320985 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.321032 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.320992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:48.321187 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.321138 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache" (OuterVolumeSpecName: "model-cache") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:48.321235 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.321210 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 20:26:48.323071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.323036 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 20:26:48.323071 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.323052 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2" (OuterVolumeSpecName: "kube-api-access-7rfz2") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "kube-api-access-7rfz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:26:48.323231 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.323197 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm" (OuterVolumeSpecName: "dshm") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:48.346334 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.346255 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39fab4fb-0778-4034-b89f-5c8ced6c665d" (UID: "39fab4fb-0778-4034-b89f-5c8ced6c665d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:26:48.407086 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407052 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-994b97cc4-8nphm_39fab4fb-0778-4034-b89f-5c8ced6c665d/main/0.log" Sep 16 20:26:48.407453 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407430 2565 generic.go:356] "Generic (PLEG): container finished" podID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerID="c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859" exitCode=137 Sep 16 20:26:48.407542 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerDied","Data":"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859"} Sep 16 20:26:48.407542 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407490 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" Sep 16 20:26:48.407542 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm" event={"ID":"39fab4fb-0778-4034-b89f-5c8ced6c665d","Type":"ContainerDied","Data":"7c62fdc40e0ce70ce299c0b5aa4544831501a7b257ef28758682994024c4ed90"} Sep 16 20:26:48.407645 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.407550 2565 scope.go:117] "RemoveContainer" containerID="c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859" Sep 16 20:26:48.417760 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.417742 2565 scope.go:117] "RemoveContainer" containerID="63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3" Sep 16 20:26:48.421634 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421611 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rfz2\" (UniqueName: \"kubernetes.io/projected/39fab4fb-0778-4034-b89f-5c8ced6c665d-kube-api-access-7rfz2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421636 2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-kserve-provision-location\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421648 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fab4fb-0778-4034-b89f-5c8ced6c665d-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421662 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421673 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421685 2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39fab4fb-0778-4034-b89f-5c8ced6c665d-cabundle-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.421743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.421696 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fab4fb-0778-4034-b89f-5c8ced6c665d-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:26:48.426261 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.426244 2565 scope.go:117] "RemoveContainer" containerID="c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859" Sep 16 20:26:48.426564 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:26:48.426545 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859\": container with ID starting with c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859 not found: ID does not exist" containerID="c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859" Sep 16 20:26:48.426618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.426572 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859"} err="failed to get container status \"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859\": rpc error: code = NotFound desc = could not find container \"c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859\": container with ID starting with c3780fba6369fb7a92a8823704ad8d0535a9146762955bd15c84dafb12e83859 not found: ID does not exist" Sep 16 20:26:48.426618 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.426589 2565 scope.go:117] "RemoveContainer" containerID="63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3" Sep 16 20:26:48.426846 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:26:48.426829 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3\": container with ID starting with 63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3 not found: ID does not exist" containerID="63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3" Sep 16 20:26:48.426885 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.426852 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3"} err="failed to get container status \"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3\": rpc error: code = NotFound desc = could not find container \"63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3\": container with ID starting with 63405465dfc5d4ca4a784acc589a91f067575253f64e735c3c30c1d45a74c8b3 not found: ID does not exist" Sep 16 20:26:48.432311 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.431570 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:26:48.438754 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:48.438731 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-994b97cc4-8nphm"] Sep 16 20:26:50.254257 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:50.254220 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" path="/var/lib/kubelet/pods/39fab4fb-0778-4034-b89f-5c8ced6c665d/volumes" Sep 16 20:26:53.518208 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518159 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6z845/must-gather-fcmc7"] Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518622 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518636 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518649 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="storage-initializer" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518655 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="storage-initializer" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518670 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerName="main" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518675 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerName="main" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518733 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cbc780-a5be-4823-8fce-afe003c2fc77" containerName="main" Sep 16 20:26:53.518804 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.518740 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="39fab4fb-0778-4034-b89f-5c8ced6c665d" containerName="main" Sep 16 20:26:53.523454 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.523432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.525749 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.525728 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6z845\"/\"default-dockercfg-pgxls\"" Sep 16 20:26:53.526616 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.526594 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6z845\"/\"kube-root-ca.crt\"" Sep 16 20:26:53.526726 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.526689 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6z845\"/\"openshift-service-ca.crt\"" Sep 16 20:26:53.530335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.530309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6z845/must-gather-fcmc7"] Sep 16 20:26:53.666807 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.666772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.666989 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.666834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghc9z\" (UniqueName: \"kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.767856 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.767822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc9z\" (UniqueName: \"kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.768029 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.767895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.768204 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.768189 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.778305 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.778213 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc9z\" (UniqueName: \"kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z\") pod \"must-gather-fcmc7\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:53.833968 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:53.833933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:26:54.107113 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:54.106970 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6z845/must-gather-fcmc7"] Sep 16 20:26:54.110073 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:26:54.110031 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74b0531d_fe4f_472b_81e5_68b3a86b9cf0.slice/crio-21d265012d2c0755e6bbcee0d231a7b61736549bef356983798179cfa761387b WatchSource:0}: Error finding container 21d265012d2c0755e6bbcee0d231a7b61736549bef356983798179cfa761387b: Status 404 returned error can't find the container with id 21d265012d2c0755e6bbcee0d231a7b61736549bef356983798179cfa761387b Sep 16 20:26:54.111904 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:54.111889 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 20:26:54.442347 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:26:54.442249 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6z845/must-gather-fcmc7" event={"ID":"74b0531d-fe4f-472b-81e5-68b3a86b9cf0","Type":"ContainerStarted","Data":"21d265012d2c0755e6bbcee0d231a7b61736549bef356983798179cfa761387b"} Sep 16 20:27:01.500242 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:01.500202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6z845/must-gather-fcmc7" event={"ID":"74b0531d-fe4f-472b-81e5-68b3a86b9cf0","Type":"ContainerStarted","Data":"b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe"} Sep 16 20:27:01.500242 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:01.500248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6z845/must-gather-fcmc7" event={"ID":"74b0531d-fe4f-472b-81e5-68b3a86b9cf0","Type":"ContainerStarted","Data":"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58"} Sep 16 20:27:01.520820 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:01.520760 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6z845/must-gather-fcmc7" podStartSLOduration=2.236163062 podStartE2EDuration="8.52074318s" totalImagesPullingTime="6.284580118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:01.519121602 +0000 UTC m=+4905.725077382" watchObservedRunningTime="2026-09-16 20:27:01.52074318 +0000 UTC m=+4905.726698933" Sep 16 20:27:17.103351 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:17.103320 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cf589d94b-rxcqc_70722144-5165-41ed-9e93-0c3fd3354607/router/0.log" Sep 16 20:27:18.030373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:18.030342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cf589d94b-rxcqc_70722144-5165-41ed-9e93-0c3fd3354607/router/0.log" Sep 16 20:27:19.562845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:19.562810 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vx52f_f147649b-57d6-4b35-85b7-3a5a5db2f626/manager/0.log" Sep 16 20:27:19.572064 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:19.572040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xpwcg_1c1c8572-4101-4733-8d39-83b2a1f1260d/limitador/0.log" Sep 16 20:27:19.586737 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:19.586699 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qhg5f_f9ce13d9-b9f4-42f4-874f-1d71c55dfb10/manager/0.log" Sep 16 20:27:22.651683 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:22.651646 2565 generic.go:356] "Generic (PLEG): container finished" podID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerID="2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58" exitCode=0 Sep 16 20:27:22.652191 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:22.651723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6z845/must-gather-fcmc7" event={"ID":"74b0531d-fe4f-472b-81e5-68b3a86b9cf0","Type":"ContainerDied","Data":"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58"} Sep 16 20:27:22.652191 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:22.652105 2565 scope.go:117] "RemoveContainer" containerID="2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58" Sep 16 20:27:23.604840 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:23.604777 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6z845_must-gather-fcmc7_74b0531d-fe4f-472b-81e5-68b3a86b9cf0/gather/0.log" Sep 16 20:27:24.211941 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.211902 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vcftn/must-gather-4ltsz"] Sep 16 20:27:24.215192 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.215158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.217739 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.217709 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vcftn\"/\"default-dockercfg-hfqdr\"" Sep 16 20:27:24.217873 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.217742 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"kube-root-ca.crt\"" Sep 16 20:27:24.217873 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.217829 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vcftn\"/\"openshift-service-ca.crt\"" Sep 16 20:27:24.224753 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.224731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/must-gather-4ltsz"] Sep 16 20:27:24.357125 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.357087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjvv\" (UniqueName: \"kubernetes.io/projected/a0abc844-a818-41bd-9d53-65d58ce2a994-kube-api-access-jwjvv\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.357333 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.357141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0abc844-a818-41bd-9d53-65d58ce2a994-must-gather-output\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.457953 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.457912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjvv\" (UniqueName: \"kubernetes.io/projected/a0abc844-a818-41bd-9d53-65d58ce2a994-kube-api-access-jwjvv\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.458134 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.457972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0abc844-a818-41bd-9d53-65d58ce2a994-must-gather-output\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.458341 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.458325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a0abc844-a818-41bd-9d53-65d58ce2a994-must-gather-output\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.467743 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.467720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjvv\" (UniqueName: \"kubernetes.io/projected/a0abc844-a818-41bd-9d53-65d58ce2a994-kube-api-access-jwjvv\") pod \"must-gather-4ltsz\" (UID: \"a0abc844-a818-41bd-9d53-65d58ce2a994\") " pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.525373 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.525330 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/must-gather-4ltsz" Sep 16 20:27:24.801249 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:24.801224 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/must-gather-4ltsz"] Sep 16 20:27:24.803373 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:27:24.803346 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0abc844_a818_41bd_9d53_65d58ce2a994.slice/crio-cc678c77abc0fb85f3a2f22bf6c73344d2cc41e25895578ff7a0efea1b8f9557 WatchSource:0}: Error finding container cc678c77abc0fb85f3a2f22bf6c73344d2cc41e25895578ff7a0efea1b8f9557: Status 404 returned error can't find the container with id cc678c77abc0fb85f3a2f22bf6c73344d2cc41e25895578ff7a0efea1b8f9557 Sep 16 20:27:25.671915 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:25.671865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/must-gather-4ltsz" event={"ID":"a0abc844-a818-41bd-9d53-65d58ce2a994","Type":"ContainerStarted","Data":"cc678c77abc0fb85f3a2f22bf6c73344d2cc41e25895578ff7a0efea1b8f9557"} Sep 16 20:27:26.683126 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:26.683085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/must-gather-4ltsz" event={"ID":"a0abc844-a818-41bd-9d53-65d58ce2a994","Type":"ContainerStarted","Data":"83b993bed36cb85359c323fd1f8de8028e87a6509e2cc344807b0ecdd1399fdb"} Sep 16 20:27:26.683694 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:26.683462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/must-gather-4ltsz" event={"ID":"a0abc844-a818-41bd-9d53-65d58ce2a994","Type":"ContainerStarted","Data":"bc3a17ee501564dff3bb05e142bd33a36f7f5c385c3c00363a02b33240d49eef"} Sep 16 20:27:26.700907 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:26.700845 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vcftn/must-gather-4ltsz" podStartSLOduration=1.8884688920000001 podStartE2EDuration="2.700812522s" totalImagesPullingTime="812.34363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:26.699220144 +0000 UTC m=+4930.905175906" watchObservedRunningTime="2026-09-16 20:27:26.700812522 +0000 UTC m=+4930.906768277" Sep 16 20:27:27.293484 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:27.293442 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cpgbn_13a5cf8d-5203-4fd9-b4cc-03d2d8a0c830/global-pull-secret-syncer/0.log" Sep 16 20:27:27.435845 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:27.435810 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkrhj_57f5c18c-6c8b-4cc8-b36f-67dba09488e7/konnectivity-agent/0.log" Sep 16 20:27:27.518002 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:27.517974 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-148.ec2.internal_f1353260c7ace9243a51756424e4b1aa/haproxy/0.log" Sep 16 20:27:29.063237 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.062409 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6z845/must-gather-fcmc7"] Sep 16 20:27:29.063237 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.062641 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6z845/must-gather-fcmc7" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="copy" containerID="cri-o://b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe" gracePeriod=2 Sep 16 20:27:29.065008 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.064976 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcmc7\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6z845\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.066104 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.066070 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6z845/must-gather-fcmc7"] Sep 16 20:27:29.435163 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.435137 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6z845_must-gather-fcmc7_74b0531d-fe4f-472b-81e5-68b3a86b9cf0/copy/0.log" Sep 16 20:27:29.438465 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.438439 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.441530 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.441498 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcmc7\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6z845\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.516311 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.515737 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output\") pod \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " Sep 16 20:27:29.516311 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.515915 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghc9z\" (UniqueName: \"kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z\") pod \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\" (UID: \"74b0531d-fe4f-472b-81e5-68b3a86b9cf0\") " Sep 16 20:27:29.521212 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.519829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z" (OuterVolumeSpecName: "kube-api-access-ghc9z") pod "74b0531d-fe4f-472b-81e5-68b3a86b9cf0" (UID: "74b0531d-fe4f-472b-81e5-68b3a86b9cf0"). InnerVolumeSpecName "kube-api-access-ghc9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 20:27:29.529722 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.529487 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "74b0531d-fe4f-472b-81e5-68b3a86b9cf0" (UID: "74b0531d-fe4f-472b-81e5-68b3a86b9cf0"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 20:27:29.617220 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.617173 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghc9z\" (UniqueName: \"kubernetes.io/projected/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-kube-api-access-ghc9z\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:27:29.617220 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.617224 2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74b0531d-fe4f-472b-81e5-68b3a86b9cf0-must-gather-output\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\"" Sep 16 20:27:29.711749 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.711665 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6z845_must-gather-fcmc7_74b0531d-fe4f-472b-81e5-68b3a86b9cf0/copy/0.log" Sep 16 20:27:29.712627 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.712410 2565 generic.go:356] "Generic (PLEG): container finished" podID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerID="b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe" exitCode=143 Sep 16 20:27:29.712627 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.712475 2565 scope.go:117] "RemoveContainer" containerID="b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe" Sep 16 20:27:29.713216 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.713187 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.715707 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.715652 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcmc7\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6z845\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.727003 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.726673 2565 scope.go:117] "RemoveContainer" containerID="2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58" Sep 16 20:27:29.741198 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.741164 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fcmc7\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6z845\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" pod="openshift-must-gather-6z845/must-gather-fcmc7" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.764620 2565 scope.go:117] "RemoveContainer" containerID="b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.765081 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe\": container with ID starting with b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe not found: ID does not exist" containerID="b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.765121 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe"} err="failed to get container status \"b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe\": rpc error: code = NotFound desc = could not find container \"b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe\": container with ID starting with b37db79cbc1c84567c2510448f99089549b9505e627ce1fa7b194e7074c23cbe not found: ID does not exist" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.765169 2565 scope.go:117] "RemoveContainer" containerID="2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: E0916 20:27:29.765483 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58\": container with ID starting with 2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58 not found: ID does not exist" containerID="2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58" Sep 16 20:27:29.769335 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:29.765511 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58"} err="failed to get container status \"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58\": rpc error: code = NotFound desc = could not find container \"2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58\": container with ID starting with 2e71cb95d40b9d4fdd73f07641eeb1c2de9a9aad83cef482cfd89a168b8afc58 not found: ID does not exist" Sep 16 20:27:30.261253 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:30.261219 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" path="/var/lib/kubelet/pods/74b0531d-fe4f-472b-81e5-68b3a86b9cf0/volumes" Sep 16 20:27:33.023436 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:33.023380 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vx52f_f147649b-57d6-4b35-85b7-3a5a5db2f626/manager/0.log" Sep 16 20:27:33.047293 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:33.047240 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-xpwcg_1c1c8572-4101-4733-8d39-83b2a1f1260d/limitador/0.log" Sep 16 20:27:33.079650 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:33.079616 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qhg5f_f9ce13d9-b9f4-42f4-874f-1d71c55dfb10/manager/0.log" Sep 16 20:27:34.698079 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:34.698041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mxbx6_2537aaa2-0a6f-4c57-875b-aeb59cbf5555/kube-state-metrics/0.log" Sep 16 20:27:34.727966 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:34.727935 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mxbx6_2537aaa2-0a6f-4c57-875b-aeb59cbf5555/kube-rbac-proxy-main/0.log" Sep 16 20:27:34.754817 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:34.754768 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-mxbx6_2537aaa2-0a6f-4c57-875b-aeb59cbf5555/kube-rbac-proxy-self/0.log" Sep 16 20:27:35.151778 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.151744 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzw2w_40643aef-5885-4b34-9d3c-3e3b3e34037d/node-exporter/0.log" Sep 16 20:27:35.193800 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.193767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzw2w_40643aef-5885-4b34-9d3c-3e3b3e34037d/kube-rbac-proxy/0.log" Sep 16 20:27:35.221995 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.221971 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzw2w_40643aef-5885-4b34-9d3c-3e3b3e34037d/init-textfile/0.log" Sep 16 20:27:35.264062 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.264031 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw8vl_c23a1749-534f-490b-8acf-74223843a16f/kube-rbac-proxy-main/0.log" Sep 16 20:27:35.300589 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.300561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw8vl_c23a1749-534f-490b-8acf-74223843a16f/kube-rbac-proxy-self/0.log" Sep 16 20:27:35.333398 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.333313 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw8vl_c23a1749-534f-490b-8acf-74223843a16f/openshift-state-metrics/0.log" Sep 16 20:27:35.614995 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.614906 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mxzsm_b1dad97e-78e6-4f43-9f82-19335c54019d/prometheus-operator/0.log" Sep 16 20:27:35.633860 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.633831 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mxzsm_b1dad97e-78e6-4f43-9f82-19335c54019d/kube-rbac-proxy/0.log" Sep 16 20:27:35.704819 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.704783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548845546f-2tvlf_a968364d-0f11-4903-8d76-2e8af58920ae/telemeter-client/0.log" Sep 16 20:27:35.732466 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.732436 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548845546f-2tvlf_a968364d-0f11-4903-8d76-2e8af58920ae/reload/0.log" Sep 16 20:27:35.756784 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.756753 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-548845546f-2tvlf_a968364d-0f11-4903-8d76-2e8af58920ae/kube-rbac-proxy/0.log" Sep 16 20:27:35.802515 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.802489 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/thanos-query/0.log" Sep 16 20:27:35.831096 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.831064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/kube-rbac-proxy-web/0.log" Sep 16 20:27:35.857577 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.857547 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/kube-rbac-proxy/0.log" Sep 16 20:27:35.890486 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.890404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/prom-label-proxy/0.log" Sep 16 20:27:35.916634 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.916589 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/kube-rbac-proxy-rules/0.log" Sep 16 20:27:35.941601 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:35.941567 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8568f88679-mblvk_fd3a7fa7-a423-4f4e-bc2f-8f3488460dfe/kube-rbac-proxy-metrics/0.log" Sep 16 20:27:37.067460 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.067434 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-dpzbq_2272dae0-6797-4da6-a872-9b1e8fc7711c/networking-console-plugin/0.log" Sep 16 20:27:37.579212 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579173 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn"] Sep 16 20:27:37.579659 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579642 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="copy" Sep 16 20:27:37.579659 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579658 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="copy" Sep 16 20:27:37.579814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579684 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="gather" Sep 16 20:27:37.579814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579690 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="gather" Sep 16 20:27:37.579814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579766 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="gather" Sep 16 20:27:37.579814 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.579774 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b0531d-fe4f-472b-81e5-68b3a86b9cf0" containerName="copy" Sep 16 20:27:37.584339 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.584314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.592533 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.592501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn"] Sep 16 20:27:37.709914 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.709873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-lib-modules\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.710263 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.710236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-sys\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.710475 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.710458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-proc\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.710668 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.710652 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-podres\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.710836 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.710818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw8c\" (UniqueName: \"kubernetes.io/projected/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-kube-api-access-hhw8c\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811646 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-lib-modules\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-sys\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-proc\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-podres\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw8c\" (UniqueName: \"kubernetes.io/projected/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-kube-api-access-hhw8c\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-sys\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-lib-modules\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.811825 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-proc\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.812075 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.811863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-podres\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.820081 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.820057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw8c\" (UniqueName: \"kubernetes.io/projected/947d37a9-8384-4c95-91d6-a6bd8ed2cc9e-kube-api-access-hhw8c\") pod \"perf-node-gather-daemonset-z9tpn\" (UID: \"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e\") " pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:37.900034 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:37.899960 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:38.142208 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.142171 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59cf576fd8-plg8d_df06f925-fb23-4853-9e51-aa586209fd51/console/0.log" Sep 16 20:27:38.196440 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.196404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn"] Sep 16 20:27:38.197802 ip-10-0-128-148 kubenswrapper[2565]: W0916 20:27:38.197773 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod947d37a9_8384_4c95_91d6_a6bd8ed2cc9e.slice/crio-9205df573d18000619ed4b72c506c54c0c239930f27cd655bce148cae853748a WatchSource:0}: Error finding container 9205df573d18000619ed4b72c506c54c0c239930f27cd655bce148cae853748a: Status 404 returned error can't find the container with id 9205df573d18000619ed4b72c506c54c0c239930f27cd655bce148cae853748a Sep 16 20:27:38.202461 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.202440 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-ck5md_56b360ba-98b8-4d60-b833-9a2d3f842a0d/download-server/0.log" Sep 16 20:27:38.798364 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.798324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" event={"ID":"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e","Type":"ContainerStarted","Data":"7e9c4cdea399f65f5f09048cc4f656d2480b1d1cf24b272a110ab7f15eb6477f"} Sep 16 20:27:38.798558 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.798384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" event={"ID":"947d37a9-8384-4c95-91d6-a6bd8ed2cc9e","Type":"ContainerStarted","Data":"9205df573d18000619ed4b72c506c54c0c239930f27cd655bce148cae853748a"} Sep 16 20:27:38.798558 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.798456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:38.817464 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.817412 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" podStartSLOduration=1.817395052 podStartE2EDuration="1.817395052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:27:38.815398856 +0000 UTC m=+4943.021354609" watchObservedRunningTime="2026-09-16 20:27:38.817395052 +0000 UTC m=+4943.023350805" Sep 16 20:27:38.818700 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:38.818673 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-2rb74_b75e5ca5-b5a3-4f2d-811f-3b96aff46ddc/volume-data-source-validator/0.log" Sep 16 20:27:39.809895 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:39.809864 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-969r4_8452df11-18f3-46c0-9353-920103de5013/dns/0.log" Sep 16 20:27:39.853450 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:39.853412 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-969r4_8452df11-18f3-46c0-9353-920103de5013/kube-rbac-proxy/0.log" Sep 16 20:27:40.086389 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:40.086306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9fwsh_b547de82-20e4-4a98-b580-70f04676b620/dns-node-resolver/0.log" Sep 16 20:27:40.755628 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:40.755598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj67g_297542fb-6cc8-409f-a483-4749340921ef/node-ca/0.log" Sep 16 20:27:41.983316 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:41.983264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cf589d94b-rxcqc_70722144-5165-41ed-9e93-0c3fd3354607/router/0.log" Sep 16 20:27:42.552692 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:42.552661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wvl7c_18bd5854-61a8-4b7e-a7c1-e847f7dfaa3b/serve-healthcheck-canary/0.log" Sep 16 20:27:42.997514 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:42.997478 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-5lz2w_aed24638-847e-4542-82ad-ac5eee3df193/insights-operator/0.log" Sep 16 20:27:42.999436 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:42.999417 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-5lz2w_aed24638-847e-4542-82ad-ac5eee3df193/insights-operator/1.log" Sep 16 20:27:43.088588 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:43.088561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwzlf_9dd12d57-2add-45a4-9acd-ceb8d543cb0d/kube-rbac-proxy/0.log" Sep 16 20:27:43.109999 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:43.109968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwzlf_9dd12d57-2add-45a4-9acd-ceb8d543cb0d/exporter/0.log" Sep 16 20:27:43.131251 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:43.131218 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwzlf_9dd12d57-2add-45a4-9acd-ceb8d543cb0d/extractor/0.log" Sep 16 20:27:44.817163 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:44.817128 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vcftn/perf-node-gather-daemonset-z9tpn" Sep 16 20:27:46.343182 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:46.343153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pnmgb_c328a311-dff1-4e95-a032-ec92725406f2/openshift-lws-operator/0.log" Sep 16 20:27:47.052923 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:47.052894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-58dc55f4b7-bkbbz_2fcddc6a-143b-4386-a714-08b12044088a/manager/0.log" Sep 16 20:27:47.605198 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:47.605112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-776f74ff6f-6pndf_575fda9f-5e01-4324-86e5-605727d99fd5/manager/0.log" Sep 16 20:27:47.641010 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:47.640939 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-cnxwn_4696e711-30fe-42d0-b3ed-95a7bac3b5c2/server/0.log" Sep 16 20:27:47.753668 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:47.753640 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cf589d94b-rxcqc_70722144-5165-41ed-9e93-0c3fd3354607/router/0.log" Sep 16 20:27:48.886395 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:48.886343 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-bgs5l_5d17a5f4-a60b-47ef-99c8-128f4a388f4e/manager/0.log" Sep 16 20:27:48.922436 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:48.922409 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qd7tm_8a61edf7-9fe0-438e-a59a-47bc6fe3ce64/s3-init/0.log" Sep 16 20:27:48.940140 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:48.940106 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qd7tm_8a61edf7-9fe0-438e-a59a-47bc6fe3ce64/download-hf-model/0.log" Sep 16 20:27:48.972978 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:48.972921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vd66v_490a708e-ac96-4ed3-9aeb-4024ddf1ec2d/seaweedfs/0.log" Sep 16 20:27:49.203516 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:49.203443 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cf589d94b-rxcqc_70722144-5165-41ed-9e93-0c3fd3354607/router/0.log" Sep 16 20:27:54.984095 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:54.984066 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/kube-multus-additional-cni-plugins/0.log" Sep 16 20:27:55.005687 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.005656 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/egress-router-binary-copy/0.log" Sep 16 20:27:55.026694 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.026667 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/cni-plugins/0.log" Sep 16 20:27:55.047533 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.047505 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/bond-cni-plugin/0.log" Sep 16 20:27:55.068410 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.068381 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/routeoverride-cni/0.log" Sep 16 20:27:55.088643 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.088620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/whereabouts-cni-bincopy/0.log" Sep 16 20:27:55.109299 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.109252 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fp9xj_5da48f5d-46b1-4fe4-b6e8-3aa0640f74de/whereabouts-cni/0.log" Sep 16 20:27:55.514926 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.514890 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-czckf_d27b7496-86c5-4818-89b4-8d50a41259bc/kube-multus/0.log" Sep 16 20:27:55.643703 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.643666 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vnwn8_b4ea1abe-95f7-4dbc-b130-c3b9b471452a/network-metrics-daemon/0.log" Sep 16 20:27:55.662677 ip-10-0-128-148 kubenswrapper[2565]: I0916 20:27:55.662643 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vnwn8_b4ea1abe-95f7-4dbc-b130-c3b9b471452a/kube-rbac-proxy/0.log"