Sep 17 19:03:58.783903 ip-10-0-132-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 19:03:58.783914 ip-10-0-132-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 19:03:58.783922 ip-10-0-132-57 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 19:03:58.784226 ip-10-0-132-57 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 19:04:08.516373 ip-10-0-132-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 19:04:08.516386 ip-10-0-132-57 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 889b828ea08a41a29baf406c8a61ab0c --
Sep 17 19:06:06.511678 ip-10-0-132-57 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:09.970275 ip-10-0-132-57 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:09.970275 ip-10-0-132-57 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:09.970275 ip-10-0-132-57 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:09.970275 ip-10-0-132-57 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:09.970275 ip-10-0-132-57 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:09.970968 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.970342    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:09.974074 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974059    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.974074 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974074    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974078    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974081    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974086    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974090    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974094    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974097    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974099    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974102    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974105    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974114    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974117    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974120    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974122    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974125    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974128    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974130    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974133    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974135    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.974135 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974138    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974141    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974143    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974146    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974149    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974151    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974154    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974156    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974159    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974161    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974165    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974169    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974172    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974175    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974178    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974180    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974183    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974185    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974188    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974190    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.974601 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974193    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974195    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974199    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974202    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974204    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974207    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974209    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974211    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974214    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974216    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974219    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974223    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974226    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974229    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974232    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974235    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974238    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974241    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974243    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.975079 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974246    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974249    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974251    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974254    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974256    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974259    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974261    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974264    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974266    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974269    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974271    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974274    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974276    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974279    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974281    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974284    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974286    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974289    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974292    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974294    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.975558 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974297    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974299    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974303    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974305    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974308    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974310    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974313    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974316    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974319    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974322    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974324    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974327    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974329    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974332    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974334    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974337    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974339    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974342    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974344    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.974347    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.976029 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975043    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975049    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975052    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975055    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975058    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975060    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975063    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975066    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975069    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975071    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975074    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975077    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975080    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975082    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975085    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975088    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975091    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975094    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975098    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.976541 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975103    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975106    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975109    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975112    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975115    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975119    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975122    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975125    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975131    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975134    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975136    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975139    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975142    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975145    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975148    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975150    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975153    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975156    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975160    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.977009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975163    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975166    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975169    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975171    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975175    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975177    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975180    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975182    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975185    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975187    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975190    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975192    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975197    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975201    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975204    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975206    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975209    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975211    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975214    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.977521 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975216    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975218    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975221    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975223    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975226    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975228    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975232    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975234    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975237    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975239    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975242    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975244    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975247    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975249    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975252    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975254    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975257    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975259    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975262    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975265    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.977978 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975269    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975272    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975274    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975277    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975282    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975284    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975287    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975290    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975292    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975295    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975297    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975300    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975302    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975304    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975307    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975310    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975312    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975315    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975317    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.978463 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975320    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975693    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975698    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975701    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975703    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975706    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975709    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975711    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975714    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975716    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975719    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975722    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975724    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975727    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975729    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975732    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975735    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975738    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975741    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975744    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.978969 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975746    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975749    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975751    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975754    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975758    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975762    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975766    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975769    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975773    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975776    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975778    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975781    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975783    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975786    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975788    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975791    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975793    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975796    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975798    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.979472 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975801    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975804    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975806    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975809    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975812    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975814    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975817    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975819    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975823    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975826    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975829    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975831    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975834    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975836    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975839    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975842    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975844    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975847    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975849    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975852    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.979968 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975856    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975859    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975861    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975864    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975866    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975869    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975871    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975873    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975876    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975878    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975881    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975883    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975886    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975890    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975893    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975897    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975900    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975903    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975906    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.980496 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975908    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975911    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975914    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975917    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975920    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975923    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975926    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975928    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975931    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975934    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975936    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975939    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975941    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975944    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975946    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975949    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975951    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975954    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975956    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975959    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.980945 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975961    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.975963    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976032    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976063    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976069    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976074    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976077    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976080    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976084    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976088    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976091    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976095    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976098    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976102    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976108    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976113    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976116    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976119    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976122    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976125    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976128    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976131    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976134    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976137    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:09.981502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976140    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976144    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976146    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976157    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976160    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976164    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976166    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976169    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976172    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976176    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976180    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976183    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976186    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976189    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976192    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976195    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976199    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976202    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976205    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976208    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976211    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976215    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976219    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976224    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976227    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:09.982086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976230    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976233    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976235    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976238    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976241    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976244    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976247    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976250    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976253    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976256    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976259    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976262    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976265    2581 flags.go:64] FLAG: --help="false"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976268    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-132-57.ec2.internal"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976271    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976274    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976277    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976280    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976283    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976285    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976288    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976291    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976294    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:09.982696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976297    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976300    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976302    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976305    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976308    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976311    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976316    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976320    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976323    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976326    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976328    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976333    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976336    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976339    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976342    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976345    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976348    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976351    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976353    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976358    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976361    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976364    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976367    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976370    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976373    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:09.983273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976376    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976378    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976381    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976384    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976390    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976393    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976396    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976399    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976402    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976407    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976410    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976428    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976431    2581 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976436    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976439    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0db0668d6be42b786"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976443    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976446    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976449    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976452    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976455    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976458    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976461    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976463    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976467    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976470    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:09.983953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976473    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976476    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976478    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976482    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976485    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976488    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976490    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976493    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976496    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976499    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976502    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976505    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976508    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976510    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976513    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976516    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976519    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976522    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976528    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976530    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976534    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976538    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976542    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976545    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976548    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:09.984611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976551    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976554    2581 flags.go:64] FLAG: --v="2"
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976558    2581 flags.go:64] FLAG: --version="false"
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976562    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976565    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.976568    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976683    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976687    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976690    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976693    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976696    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976699    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976702    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976705    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976708    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976711    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976714    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976717    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976721    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976725    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.985515 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976727    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976730    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976733    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976736    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976739    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976742    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976744    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976748    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976750    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976754    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976756    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976760    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976763    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976766    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976768    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976771    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976775    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976778    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976781    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.986131 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976784    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976787    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976791    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976795    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976799    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976803    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976805    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976808    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976811    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976813    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976816    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976818    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976821    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976823    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976826    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976829    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976831    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976834    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976836    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976839    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.986612 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976842    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976844    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976847    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976851    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976854    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976856    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976859    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976862    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976864    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976867    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976869    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976872    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976875    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976877    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976880    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976882    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976885    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976888    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976890    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976893    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.987103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976895    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976898    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976900    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976903    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976905    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976908    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976911    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976913    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976916    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976918    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976921    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976924    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976926    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976929    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976932    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976934    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976937    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976940    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976942    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976945    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.987615 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976947    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976950    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976952    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976955    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976957    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.976960    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.977980    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.986732    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.986747    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.986767    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.986775    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986835    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986840    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986844    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986847    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.988103 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986850    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986853    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986856    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986859    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986862    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986865    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986868    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986870    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986873    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986875    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986878    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986881    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986884    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986887    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986890    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986893    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986896    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986898    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986901    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986903    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.988505 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986907    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986911    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986915    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986918    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986921    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986926    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986930    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986933    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986935    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986938    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986941    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986943    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986947    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986949    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986952    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986955    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986958    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986960    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986963    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.989019 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986966    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986968    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986971    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986974    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986977    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986979    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986982    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986985    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986988    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986991    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986993    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986996    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.986999    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987001    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987004    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987007    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987010    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987012    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987015    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.989507 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987017    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987021    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987025    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987028    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987031    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987033    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987036    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987038    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987041    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987043    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987046    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987049    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987051    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987054    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987056    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987058    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987061    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987064    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987067    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987070    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.989965 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987073    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987075    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987078    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987081    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987083    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987086    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987088    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987091    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987093    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987095    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987098    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987100    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987103    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987105    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987108    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987111    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.990490 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987114    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.987119    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987234    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987239    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987242    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987245    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987248    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987251    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987254    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987256    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987259    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987262    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987264    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987266    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987270    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:09.990910 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987272    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987276    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987280    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987282    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987285    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987288    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987290    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987293    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987295    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987298    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987300    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987303    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987305    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987308    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987310    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987313    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987315    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987318    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987321    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:09.991282 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987323    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987326    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987328    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987331    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987333    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987335    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987338    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987340    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987343    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987345    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987348    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987351    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987353    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987357    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987359    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987362    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987365    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987367    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987370    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987372    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:09.991811 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987375    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987377    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987380    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987382    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987385    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987389    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987392    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987395    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987398    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987401    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987403    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987406    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987408    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987445    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987450    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987453    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987456    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987459    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987462    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:09.992301 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987464    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987467    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987470    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987472    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987478    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987481    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987484    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987487    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987490    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987493    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987496    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987498    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987501    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987503    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987506    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987509    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987511    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987515    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987519    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987521    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:09.992856 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987524    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987527    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987530    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987532    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987535    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987538    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987541    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:09.987543    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.987548    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.987650    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.990526    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.991360    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.991473    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:09.993334 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.991502    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:09.993782 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.993768    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:09.995030 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:09.995012    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:10.000091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.000073    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:10.001949 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.001935    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:10.003282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.003267    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:10.006062 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.006044    2581 fs.go:135] Filesystem UUIDs: map[7015234d-38d9-441d-a739-f796de41c144:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9876adf0-db85-46bb-8610-369551ab1cc0:/dev/nvme0n1p3]
Sep 17 19:06:10.006139 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.006062    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:10.010950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.010808    2581 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:10.009585152 +0000 UTC m=+2.023155305 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091885 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2771e2872eb8a0d7985dd702eb0302 SystemUUID:ec2771e2-872e-b8a0-d798-5dd702eb0302 BootID:889b828e-a08a-41a2-9baf-406c8a61ab0c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:0d:be:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:0d:be:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:5d:6e:02:c6:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:10.010950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.010945    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:10.011053 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.011015    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:10.012502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.012480    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:10.012658 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.012506    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:10.013654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.013643    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:10.013685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.013657    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:10.013685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.013665    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:10.013685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.013676    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:10.015251 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.015241    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:10.019169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019158    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:10.019389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019377    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:10.019712 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019661    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:10.019793 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019777    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:10.019839 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019814    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:10.019839 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019828    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:10.019917 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.019841    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:10.020761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.020737    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:10.021053 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.021008    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:10.024099 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.024078    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:10.025041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.025024    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:10.027966 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.027952    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:10.028011 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.027984    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:10.028137 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028128    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:10.028195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028143    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:10.028195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028148    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:10.028195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028162    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:10.028195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028170    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:10.028195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028175    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:10.028713 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028704    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:10.028750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028715    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:10.028750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028721    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:10.028750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028726    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:10.028750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028734    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:10.028750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028742    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:10.028871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028765    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:10.028871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.028770    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:10.032639 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.032622    2581 server.go:1268] "Started kubelet"
Sep 17 19:06:10.032727 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.032692    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:10.032882 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.032834    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:10.032983 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.032897    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:10.033266 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.033251    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:10.033957 ip-10-0-132-57 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:10.034493 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.034476    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:10.040729 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.040711    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:10.040799 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.040728    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:10.040799 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.040746    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:10.041166 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.041147    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:10.041676 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.041660    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:10.041751 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.041675    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:10.041799 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.041734    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.042142 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042125    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:10.042142 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042143    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:10.042274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042153    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:10.042274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042202    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:10.042274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042208    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:10.042534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042516    2581 factory.go:55] Registering systemd factory
Sep 17 19:06:10.042534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042535    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:10.042800 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042784    2581 factory.go:153] Registering CRI-O factory
Sep 17 19:06:10.042857 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042806    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:10.042912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042855    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:10.042912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042875    2581 factory.go:103] Registering Raw factory
Sep 17 19:06:10.042912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.042891    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:10.043223 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.043209    2581 manager.go:349] Starting recovery of all containers
Sep 17 19:06:10.070238 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.070204    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:10.070358 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.070331    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:10.073131 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.073103    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:10.077183 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.077164    2581 manager.go:354] Recovery completed
Sep 17 19:06:10.077737 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.076665    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-57.ec2.internal.18d6312cb64143e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-57.ec2.internal,UID:ip-10-0-132-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-57.ec2.internal,},FirstTimestamp:2026-09-17 19:06:10.032403424 +0000 UTC m=+2.045973575,LastTimestamp:2026-09-17 19:06:10.032403424 +0000 UTC m=+2.045973575,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-57.ec2.internal,}"
Sep 17 19:06:10.081551 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.081537    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.085151 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085123    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.085203 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085169    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.085203 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.085735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085721    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:10.085735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085735    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:10.085853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085725    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.085853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.085752    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:10.087288 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.087206    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-57.ec2.internal.18d6312cb96629d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-57.ec2.internal,UID:ip-10-0-132-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-57.ec2.internal,},FirstTimestamp:2026-09-17 19:06:10.085153238 +0000 UTC m=+2.098723390,LastTimestamp:2026-09-17 19:06:10.085153238 +0000 UTC m=+2.098723390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-57.ec2.internal,}"
Sep 17 19:06:10.087450 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.087434    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.087518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.087459    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.087518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.087469    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.087986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.087974    2581 policy_none.go:49] "None policy: Start"
Sep 17 19:06:10.088044 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.087993    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:10.088044 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.088006    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:10.089903 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.089890    2581 policy_none.go:47] "Start"
Sep 17 19:06:10.097549 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.097531    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2k69"
Sep 17 19:06:10.099332 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.099248    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-57.ec2.internal.18d6312cb9667b16  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-57.ec2.internal,UID:ip-10-0-132-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-57.ec2.internal,},FirstTimestamp:2026-09-17 19:06:10.085174038 +0000 UTC m=+2.098744189,LastTimestamp:2026-09-17 19:06:10.085174038 +0000 UTC m=+2.098744189,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-57.ec2.internal,}"
Sep 17 19:06:10.105525 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.105507    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2k69"
Sep 17 19:06:10.111507 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.111448    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-57.ec2.internal.18d6312cb9669c2e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-57.ec2.internal,UID:ip-10-0-132-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-57.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-57.ec2.internal,},FirstTimestamp:2026-09-17 19:06:10.08518251 +0000 UTC m=+2.098752662,LastTimestamp:2026-09-17 19:06:10.08518251 +0000 UTC m=+2.098752662,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-57.ec2.internal,}"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.130897    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.130963    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.130992    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131012    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.131045    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.131034    2581 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131060    2581 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131357    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131377    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131464    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131592    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.131601    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.132054    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.132121    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.138586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.133554    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:10.231583 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.231493    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.231583 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.231511    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"]
Sep 17 19:06:10.231744 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.231592    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.233070 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233050    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.233160 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233087    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.233160 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233097    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.233160 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233120    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.233160 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233050    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.233288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233162    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.233288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.233175    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.235586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.235572    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.235685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.235670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.235722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.235701    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.236990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.236967    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.237084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.236998    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.237084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.237002    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.237084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.237014    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.237084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.237023    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.237084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.237034    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.239195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.239180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.239259 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.239207    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:10.240034 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.240013    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:10.240130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.240044    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:10.240130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.240057    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:10.243126 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.243110    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.243220 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.243135    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-57.ec2.internal\": node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.244312 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.244294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.244368 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.244318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.244368 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.244335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b66c5394879f7702479f341fee142c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-57.ec2.internal\" (UID: \"03b66c5394879f7702479f341fee142c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.264333 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.264305    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-57.ec2.internal\" not found" node="ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.268833 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.268814    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-57.ec2.internal\" not found" node="ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.287127 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.287098    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.344569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.344684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.344684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b66c5394879f7702479f341fee142c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-57.ec2.internal\" (UID: \"03b66c5394879f7702479f341fee142c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.344684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.344684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d60026aede2bdb637006ae8e684c1f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal\" (UID: \"5d60026aede2bdb637006ae8e684c1f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.344684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.344680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b66c5394879f7702479f341fee142c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-57.ec2.internal\" (UID: \"03b66c5394879f7702479f341fee142c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.387766 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.387741    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.488367 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.488294    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.567983 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.567958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.572810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.572785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.589504 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.589299    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.689798 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.689763    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.790334 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:10.790264    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-57.ec2.internal\" not found"
Sep 17 19:06:10.866107 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.866084    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:10.942471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.942447    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.972870 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.972849    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:10.974582 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.974564    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal"
Sep 17 19:06:10.987131 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.987111    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:10.991249 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.991234    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:10.991358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.991340    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:10.991411 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:10.991386    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:11.021325 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.021306    2581 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:11.036759 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.036738    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:11.037135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.037113    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-q8lqz","openshift-dns/node-resolver-r29f5","openshift-image-registry/node-ca-v59rs","openshift-multus/multus-v8h59","openshift-multus/network-metrics-daemon-wgtr9","kube-system/global-pull-secret-syncer-gxwcj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal","openshift-multus/multus-additional-cni-plugins-ptw2z","openshift-network-diagnostics/network-check-target-vmlm6","openshift-network-operator/iptables-alerter-mggpm","openshift-ovn-kubernetes/ovnkube-node-7bl88","kube-system/konnectivity-agent-6qz84","kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"]
Sep 17 19:06:11.039662 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.039643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.041751 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.041691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.041829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.041765    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:11.043774 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.043758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.044371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.044349    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.044488 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.044453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dk7l2\""
Sep 17 19:06:11.045212 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.045192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8jv8k\""
Sep 17 19:06:11.045289 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.045222    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.045289 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.045195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.045356 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.045338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.045807 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.045789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.047904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.047883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-var-lib-kubelet\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.047913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-host\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.047940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051aa77a-c712-4c78-bb4b-e536d73b98f1-hosts-file\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.048007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.047972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abc53bb-f467-46c1-a12b-cc654a5e2a11-host\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.048007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.047992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4abc53bb-f467-46c1-a12b-cc654a5e2a11-serviceca\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.048068    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-systemd\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-run\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-sys\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwdc\" (UniqueName: \"kubernetes.io/projected/4abc53bb-f467-46c1-a12b-cc654a5e2a11-kube-api-access-7bwdc\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-kubernetes\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-tuned\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-tmp\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051aa77a-c712-4c78-bb4b-e536d73b98f1-tmp-dir\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-modprobe-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysconfig\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-lib-modules\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdmj\" (UniqueName: \"kubernetes.io/projected/65c25afd-98a8-469e-8b20-58d793b96d3d-kube-api-access-jqdmj\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gpm2r\""
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmfp\" (UniqueName: \"kubernetes.io/projected/051aa77a-c712-4c78-bb4b-e536d73b98f1-kube-api-access-wtmfp\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.048624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-conf\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.049047 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.048923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:11.049234 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.049212    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.049305 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.049221    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.049363 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.049220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.049363 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.049350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:11.050489 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.049882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6zldl\""
Sep 17 19:06:11.050489 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.050016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.050489 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.050209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:11.052083 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.050643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.052083 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.050724    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:11.053103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.053083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.055197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.055176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:11.055245 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.055234    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:11.055735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.055714    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:11.055964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.055945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ngrh4\""
Sep 17 19:06:11.056550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.056534    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:11.057336 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.057318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.059726 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.059707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.060843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.060824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cf44v\""
Sep 17 19:06:11.061124 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.061108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.061986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.061972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.063030 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.063014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:11.063919 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.063890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:11.064003 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.063926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:11.064042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064007    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:11.064042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.063890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:11.064104 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064011    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nrsff\""
Sep 17 19:06:11.064151 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.064297 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.064381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.064454 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.064798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gn5c9\""
Sep 17 19:06:11.064843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:11.064949 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.064935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:11.067815 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.067798    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:11.069023 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.069005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:11.069451 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.069437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:11.069492 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.069457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8mxc8\""
Sep 17 19:06:11.070767 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.070751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:11.108336 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.108320    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2b9k4"
Sep 17 19:06:11.111475 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.111433    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:10 +0000 UTC" deadline="2028-03-21 15:52:05.580735991 +0000 UTC"
Sep 17 19:06:11.111542 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.111478    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13220h45m54.469262086s"
Sep 17 19:06:11.119456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.119440    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2b9k4"
Sep 17 19:06:11.143249 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.143233    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:11.148835 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-kubernetes\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.148920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-netns\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.148920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.148920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:11.148920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-kubernetes\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdmj\" (UniqueName: \"kubernetes.io/projected/65c25afd-98a8-469e-8b20-58d793b96d3d-kube-api-access-jqdmj\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4abc53bb-f467-46c1-a12b-cc654a5e2a11-serviceca\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.148999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-registration-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96223c43-9157-4116-a57e-c191a14ae089-host-slash\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-host\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovn-node-metrics-cert\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-conf-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/6d6199c3-283b-463c-98dc-6884c1d0ebfd-kube-api-access-6n994\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-host\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ccbf37-d2de-40d3-89f8-0756439a2d4a-konnectivity-ca\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-systemd\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-slash\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-systemd\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-etc-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-k8s-cni-cncf-io\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-etc-kubernetes\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvpv\" (UniqueName: \"kubernetes.io/projected/152784a6-5c9f-4a87-8a4f-873c218f9f37-kube-api-access-drvpv\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhw59\" (UniqueName: \"kubernetes.io/projected/41df1f9f-c14e-41fb-bf22-ae8346861773-kube-api-access-vhw59\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-tuned\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-tmp\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-systemd-units\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cni-binary-copy\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-modprobe-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysconfig\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmfp\" (UniqueName: \"kubernetes.io/projected/051aa77a-c712-4c78-bb4b-e536d73b98f1-kube-api-access-wtmfp\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-var-lib-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-ovn\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-multus-certs\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.149990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-modprobe-d\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysconfig\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96223c43-9157-4116-a57e-c191a14ae089-iptables-alerter-script\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-systemd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-conf\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149732    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-script-lib\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8tp\" (UniqueName: \"kubernetes.io/projected/abe2d884-45cb-4b87-a658-ea74e9c8d013-kube-api-access-sl8tp\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-hostroot\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-sys\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-kubelet\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-kubelet-config\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-sys\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4abc53bb-f467-46c1-a12b-cc654a5e2a11-serviceca\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-cnibin\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.150914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.149991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vq5b\" (UniqueName: \"kubernetes.io/projected/a332c076-59cb-4fb4-be55-3216ae0e4d2a-kube-api-access-2vq5b\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-system-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-multus\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-dbus\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-sysctl-conf\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cnibin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-lib-modules\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-netd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-socket-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-kubelet\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-lib-modules\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-node-log\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-log-socket\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-env-overrides\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-run\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.151876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwdc\" (UniqueName: \"kubernetes.io/projected/4abc53bb-f467-46c1-a12b-cc654a5e2a11-kube-api-access-7bwdc\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-bin\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-netns\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-daemon-config\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-run\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051aa77a-c712-4c78-bb4b-e536d73b98f1-tmp-dir\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8g7\" (UniqueName: \"kubernetes.io/projected/96223c43-9157-4116-a57e-c191a14ae089-kube-api-access-qn8g7\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-socket-dir-parent\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-bin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-var-lib-kubelet\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051aa77a-c712-4c78-bb4b-e536d73b98f1-hosts-file\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abc53bb-f467-46c1-a12b-cc654a5e2a11-host\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.152674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-sys-fs\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/051aa77a-c712-4c78-bb4b-e536d73b98f1-hosts-file\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-config\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-device-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ccbf37-d2de-40d3-89f8-0756439a2d4a-agent-certs\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abc53bb-f467-46c1-a12b-cc654a5e2a11-host\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c25afd-98a8-469e-8b20-58d793b96d3d-var-lib-kubelet\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/051aa77a-c712-4c78-bb4b-e536d73b98f1-tmp-dir\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.150879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.152831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-etc-tuned\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.153607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.152839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c25afd-98a8-469e-8b20-58d793b96d3d-tmp\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.158445 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.158405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdmj\" (UniqueName: \"kubernetes.io/projected/65c25afd-98a8-469e-8b20-58d793b96d3d-kube-api-access-jqdmj\") pod \"tuned-q8lqz\" (UID: \"65c25afd-98a8-469e-8b20-58d793b96d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.158569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.158469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwdc\" (UniqueName: \"kubernetes.io/projected/4abc53bb-f467-46c1-a12b-cc654a5e2a11-kube-api-access-7bwdc\") pod \"node-ca-v59rs\" (UID: \"4abc53bb-f467-46c1-a12b-cc654a5e2a11\") " pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.158813 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.158795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmfp\" (UniqueName: \"kubernetes.io/projected/051aa77a-c712-4c78-bb4b-e536d73b98f1-kube-api-access-wtmfp\") pod \"node-resolver-r29f5\" (UID: \"051aa77a-c712-4c78-bb4b-e536d73b98f1\") " pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.184335 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.184318    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:11.251915 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.251896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-registration-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.251999 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.251921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96223c43-9157-4116-a57e-c191a14ae089-host-slash\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.251999 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.251937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.251999 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.251973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovn-node-metrics-cert\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.251998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-conf-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96223c43-9157-4116-a57e-c191a14ae089-host-slash\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-registration-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/6d6199c3-283b-463c-98dc-6884c1d0ebfd-kube-api-access-6n994\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ccbf37-d2de-40d3-89f8-0756439a2d4a-konnectivity-ca\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.252307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-conf-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-slash\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-slash\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-etc-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-k8s-cni-cncf-io\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-etc-kubernetes\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvpv\" (UniqueName: \"kubernetes.io/projected/152784a6-5c9f-4a87-8a4f-873c218f9f37-kube-api-access-drvpv\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-k8s-cni-cncf-io\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhw59\" (UniqueName: \"kubernetes.io/projected/41df1f9f-c14e-41fb-bf22-ae8346861773-kube-api-access-vhw59\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-etc-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-systemd-units\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cni-binary-copy\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-etc-kubernetes\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-var-lib-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-ovn\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-systemd-units\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.252575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-multus-certs\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96223c43-9157-4116-a57e-c191a14ae089-iptables-alerter-script\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-ovn\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-var-lib-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-systemd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-multus-certs\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-systemd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-run-openvswitch\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-script-lib\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8tp\" (UniqueName: \"kubernetes.io/projected/abe2d884-45cb-4b87-a658-ea74e9c8d013-kube-api-access-sl8tp\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-hostroot\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-kubelet\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-kubelet-config\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-cnibin\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.253208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vq5b\" (UniqueName: \"kubernetes.io/projected/a332c076-59cb-4fb4-be55-3216ae0e4d2a-kube-api-access-2vq5b\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-system-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-multus\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-dbus\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-kubelet-config\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.252990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ccbf37-d2de-40d3-89f8-0756439a2d4a-konnectivity-ca\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cnibin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-netd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-socket-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-kubelet\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-node-log\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cni-binary-copy\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-log-socket\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-multus\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96223c43-9157-4116-a57e-c191a14ae089-iptables-alerter-script\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-log-socket\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-cnibin\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-dbus\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.254042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-system-cni-dir\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-netd\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-env-overrides\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-hostroot\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-node-log\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-kubelet\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-cnibin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-socket-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-bin\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-script-lib\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-cni-bin\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-netns\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-daemon-config\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.254810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-kubelet\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-run-netns\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/152784a6-5c9f-4a87-8a4f-873c218f9f37-system-cni-dir\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8g7\" (UniqueName: \"kubernetes.io/projected/96223c43-9157-4116-a57e-c191a14ae089-kube-api-access-qn8g7\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-socket-dir-parent\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-bin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-sys-fs\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-config\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-device-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ccbf37-d2de-40d3-89f8-0756439a2d4a-agent-certs\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.255565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-netns\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-sys-fs\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-binary-copy\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-socket-dir-parent\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.253681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-env-overrides\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d6199c3-283b-463c-98dc-6884c1d0ebfd-multus-daemon-config\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d6199c3-283b-463c-98dc-6884c1d0ebfd-host-var-lib-cni-bin\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/152784a6-5c9f-4a87-8a4f-873c218f9f37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-etc-selinux\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-run-netns\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2d884-45cb-4b87-a658-ea74e9c8d013-device-dir\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.254562    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.254614    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:11.256333 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.254722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:11.754657724 +0000 UTC m=+3.768227886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:11.257041 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.254763    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:11.754738153 +0000 UTC m=+3.768308292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:11.257041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.254768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a332c076-59cb-4fb4-be55-3216ae0e4d2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.257041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.255149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovnkube-config\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.257041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.255323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a332c076-59cb-4fb4-be55-3216ae0e4d2a-ovn-node-metrics-cert\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.258045 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.258024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ccbf37-d2de-40d3-89f8-0756439a2d4a-agent-certs\") pod \"konnectivity-agent-6qz84\" (UID: \"80ccbf37-d2de-40d3-89f8-0756439a2d4a\") " pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.265644 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.265624    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:11.265800 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.265649    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:11.265800 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.265661    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:11.265800 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.265729    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:11.76571189 +0000 UTC m=+3.779282045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:11.266513 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.266408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8tp\" (UniqueName: \"kubernetes.io/projected/abe2d884-45cb-4b87-a658-ea74e9c8d013-kube-api-access-sl8tp\") pod \"aws-ebs-csi-driver-node-bd467\" (UID: \"abe2d884-45cb-4b87-a658-ea74e9c8d013\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.267320 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.267294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhw59\" (UniqueName: \"kubernetes.io/projected/41df1f9f-c14e-41fb-bf22-ae8346861773-kube-api-access-vhw59\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.267951 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.267934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8g7\" (UniqueName: \"kubernetes.io/projected/96223c43-9157-4116-a57e-c191a14ae089-kube-api-access-qn8g7\") pod \"iptables-alerter-mggpm\" (UID: \"96223c43-9157-4116-a57e-c191a14ae089\") " pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.268018 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.268002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/6d6199c3-283b-463c-98dc-6884c1d0ebfd-kube-api-access-6n994\") pod \"multus-v8h59\" (UID: \"6d6199c3-283b-463c-98dc-6884c1d0ebfd\") " pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.268523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.268499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvpv\" (UniqueName: \"kubernetes.io/projected/152784a6-5c9f-4a87-8a4f-873c218f9f37-kube-api-access-drvpv\") pod \"multus-additional-cni-plugins-ptw2z\" (UID: \"152784a6-5c9f-4a87-8a4f-873c218f9f37\") " pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.268606 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.268509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vq5b\" (UniqueName: \"kubernetes.io/projected/a332c076-59cb-4fb4-be55-3216ae0e4d2a-kube-api-access-2vq5b\") pod \"ovnkube-node-7bl88\" (UID: \"a332c076-59cb-4fb4-be55-3216ae0e4d2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.321911 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.321859    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d60026aede2bdb637006ae8e684c1f0.slice/crio-b31c7cf9aafb981e2540e8dcf2934c076cc857bb265d820ecd4b735630ddbf89 WatchSource:0}: Error finding container b31c7cf9aafb981e2540e8dcf2934c076cc857bb265d820ecd4b735630ddbf89: Status 404 returned error can't find the container with id b31c7cf9aafb981e2540e8dcf2934c076cc857bb265d820ecd4b735630ddbf89
Sep 17 19:06:11.322996 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.322980    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b66c5394879f7702479f341fee142c.slice/crio-29a8618dc3fa634e6c2383b55886c5a255dcba00cfc3533314f502fe92ef6edd WatchSource:0}: Error finding container 29a8618dc3fa634e6c2383b55886c5a255dcba00cfc3533314f502fe92ef6edd: Status 404 returned error can't find the container with id 29a8618dc3fa634e6c2383b55886c5a255dcba00cfc3533314f502fe92ef6edd
Sep 17 19:06:11.327072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.327056    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:11.363854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.363838    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:11.378943 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.378927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8lqz"
Sep 17 19:06:11.387207 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.387189    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c25afd_98a8_469e_8b20_58d793b96d3d.slice/crio-87b143bc748ffdf0d677211cf80f42f08dcc352bd5c616246776d744d1bfc96c WatchSource:0}: Error finding container 87b143bc748ffdf0d677211cf80f42f08dcc352bd5c616246776d744d1bfc96c: Status 404 returned error can't find the container with id 87b143bc748ffdf0d677211cf80f42f08dcc352bd5c616246776d744d1bfc96c
Sep 17 19:06:11.393206 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.393191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r29f5"
Sep 17 19:06:11.400820 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.400802    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051aa77a_c712_4c78_bb4b_e536d73b98f1.slice/crio-7d78aa7b5b28b113b1ccd8f109d3547b2a6cbd247257141f8f8f6aa38df32630 WatchSource:0}: Error finding container 7d78aa7b5b28b113b1ccd8f109d3547b2a6cbd247257141f8f8f6aa38df32630: Status 404 returned error can't find the container with id 7d78aa7b5b28b113b1ccd8f109d3547b2a6cbd247257141f8f8f6aa38df32630
Sep 17 19:06:11.401349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.401334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v59rs"
Sep 17 19:06:11.411184 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.411167    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abc53bb_f467_46c1_a12b_cc654a5e2a11.slice/crio-ac05ccd7fda4bdaab032de705710e394f6b902471e1dd32a9ed6396020f74d36 WatchSource:0}: Error finding container ac05ccd7fda4bdaab032de705710e394f6b902471e1dd32a9ed6396020f74d36: Status 404 returned error can't find the container with id ac05ccd7fda4bdaab032de705710e394f6b902471e1dd32a9ed6396020f74d36
Sep 17 19:06:11.420073 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.420058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8h59"
Sep 17 19:06:11.427240 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.427221    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6199c3_283b_463c_98dc_6884c1d0ebfd.slice/crio-f1c9b49d9a65761b939a85d58dc7b8e56f5e0959b7ba342ac0ffc05765fa1bf5 WatchSource:0}: Error finding container f1c9b49d9a65761b939a85d58dc7b8e56f5e0959b7ba342ac0ffc05765fa1bf5: Status 404 returned error can't find the container with id f1c9b49d9a65761b939a85d58dc7b8e56f5e0959b7ba342ac0ffc05765fa1bf5
Sep 17 19:06:11.434115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.434096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ptw2z"
Sep 17 19:06:11.441375 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.441357    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152784a6_5c9f_4a87_8a4f_873c218f9f37.slice/crio-e88267d92dfa3f6a18031016f410a25039c5543140604f69e4fa7665d433503f WatchSource:0}: Error finding container e88267d92dfa3f6a18031016f410a25039c5543140604f69e4fa7665d433503f: Status 404 returned error can't find the container with id e88267d92dfa3f6a18031016f410a25039c5543140604f69e4fa7665d433503f
Sep 17 19:06:11.452089 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.452072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mggpm"
Sep 17 19:06:11.459221 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.459202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96223c43_9157_4116_a57e_c191a14ae089.slice/crio-862e1d0871dc9d8b87016240b8d3879312c553b28efeda5e91b4854c9e313852 WatchSource:0}: Error finding container 862e1d0871dc9d8b87016240b8d3879312c553b28efeda5e91b4854c9e313852: Status 404 returned error can't find the container with id 862e1d0871dc9d8b87016240b8d3879312c553b28efeda5e91b4854c9e313852
Sep 17 19:06:11.478640 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.478623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:11.485970 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.485951    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda332c076_59cb_4fb4_be55_3216ae0e4d2a.slice/crio-255a5c9792063dccf993474593d3cf2fec2f7c4383f9c2c551417c233168cb22 WatchSource:0}: Error finding container 255a5c9792063dccf993474593d3cf2fec2f7c4383f9c2c551417c233168cb22: Status 404 returned error can't find the container with id 255a5c9792063dccf993474593d3cf2fec2f7c4383f9c2c551417c233168cb22
Sep 17 19:06:11.497089 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.497071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:11.501720 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.501706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467"
Sep 17 19:06:11.505444 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.505410    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ccbf37_d2de_40d3_89f8_0756439a2d4a.slice/crio-cec21581611b0cb027936726f011122f57cf5cd958d9aa6271207a0c570d0ebd WatchSource:0}: Error finding container cec21581611b0cb027936726f011122f57cf5cd958d9aa6271207a0c570d0ebd: Status 404 returned error can't find the container with id cec21581611b0cb027936726f011122f57cf5cd958d9aa6271207a0c570d0ebd
Sep 17 19:06:11.509579 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:11.509545    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe2d884_45cb_4b87_a658_ea74e9c8d013.slice/crio-8a1110cd28e68a8cdb9ea221a09f1917c0b19e67c6bcd412eb2f3c5636c62366 WatchSource:0}: Error finding container 8a1110cd28e68a8cdb9ea221a09f1917c0b19e67c6bcd412eb2f3c5636c62366: Status 404 returned error can't find the container with id 8a1110cd28e68a8cdb9ea221a09f1917c0b19e67c6bcd412eb2f3c5636c62366
Sep 17 19:06:11.511523 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.511504    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-bd467_openshift-cluster-csi-drivers(abe2d884-45cb-4b87-a658-ea74e9c8d013): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 19:06:11.512170 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.512147    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-bd467_openshift-cluster-csi-drivers(abe2d884-45cb-4b87-a658-ea74e9c8d013): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 19:06:11.514276 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.514260    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-bd467_openshift-cluster-csi-drivers(abe2d884-45cb-4b87-a658-ea74e9c8d013): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 19:06:11.514317 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.514293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" podUID="abe2d884-45cb-4b87-a658-ea74e9c8d013"
Sep 17 19:06:11.756835 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.756784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:11.756835 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.756818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:11.756982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.756912    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:11.756982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.756946    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:11.756982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.756965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:12.756949429 +0000 UTC m=+4.770519567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:11.757096 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.756990    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:12.756975762 +0000 UTC m=+4.770545915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:11.857314 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:11.857221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:11.857383 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.857362    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:11.857383 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.857382    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:11.857505 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.857394    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:11.857505 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:11.857468    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:12.857451361 +0000 UTC m=+4.871021516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:12.122575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.122460    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:11 +0000 UTC" deadline="2028-05-03 07:19:58.751885747 +0000 UTC"
Sep 17 19:06:12.122575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.122527    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14244h13m46.629362695s"
Sep 17 19:06:12.126241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.126217    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:12.142640 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.142547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8lqz" event={"ID":"65c25afd-98a8-469e-8b20-58d793b96d3d","Type":"ContainerStarted","Data":"87b143bc748ffdf0d677211cf80f42f08dcc352bd5c616246776d744d1bfc96c"}
Sep 17 19:06:12.150685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.150617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal" event={"ID":"03b66c5394879f7702479f341fee142c","Type":"ContainerStarted","Data":"29a8618dc3fa634e6c2383b55886c5a255dcba00cfc3533314f502fe92ef6edd"}
Sep 17 19:06:12.153220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.153159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal" event={"ID":"5d60026aede2bdb637006ae8e684c1f0","Type":"ContainerStarted","Data":"b31c7cf9aafb981e2540e8dcf2934c076cc857bb265d820ecd4b735630ddbf89"}
Sep 17 19:06:12.168557 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.164818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" event={"ID":"abe2d884-45cb-4b87-a658-ea74e9c8d013","Type":"ContainerStarted","Data":"8a1110cd28e68a8cdb9ea221a09f1917c0b19e67c6bcd412eb2f3c5636c62366"}
Sep 17 19:06:12.173031 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.172972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"255a5c9792063dccf993474593d3cf2fec2f7c4383f9c2c551417c233168cb22"}
Sep 17 19:06:12.174251 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.174150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d3869c161a073dfbaf036364ff0996f20e4d9ee240af9f0daa929f78c7ea46e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:54fc81b7047fca86ea91318cae087e3f654cc8d54725953ea6f0ddfcaf515545\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c43eb8f4e3024c90d819ecdc6c4101d91bd5c368d5b50945756ddf6fa4b3b9cf\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" podUID="abe2d884-45cb-4b87-a658-ea74e9c8d013"
Sep 17 19:06:12.187002 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.186770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerStarted","Data":"e88267d92dfa3f6a18031016f410a25039c5543140604f69e4fa7665d433503f"}
Sep 17 19:06:12.196456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.196432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8h59" event={"ID":"6d6199c3-283b-463c-98dc-6884c1d0ebfd","Type":"ContainerStarted","Data":"f1c9b49d9a65761b939a85d58dc7b8e56f5e0959b7ba342ac0ffc05765fa1bf5"}
Sep 17 19:06:12.208797 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.208067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r29f5" event={"ID":"051aa77a-c712-4c78-bb4b-e536d73b98f1","Type":"ContainerStarted","Data":"7d78aa7b5b28b113b1ccd8f109d3547b2a6cbd247257141f8f8f6aa38df32630"}
Sep 17 19:06:12.213399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.213343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qz84" event={"ID":"80ccbf37-d2de-40d3-89f8-0756439a2d4a","Type":"ContainerStarted","Data":"cec21581611b0cb027936726f011122f57cf5cd958d9aa6271207a0c570d0ebd"}
Sep 17 19:06:12.223338 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.223267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mggpm" event={"ID":"96223c43-9157-4116-a57e-c191a14ae089","Type":"ContainerStarted","Data":"862e1d0871dc9d8b87016240b8d3879312c553b28efeda5e91b4854c9e313852"}
Sep 17 19:06:12.241135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.240618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v59rs" event={"ID":"4abc53bb-f467-46c1-a12b-cc654a5e2a11","Type":"ContainerStarted","Data":"ac05ccd7fda4bdaab032de705710e394f6b902471e1dd32a9ed6396020f74d36"}
Sep 17 19:06:12.764806 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.764771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:12.764982 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.764816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:12.764982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.764909    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:12.764982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.764919    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:12.764982 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.764962    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.764945658 +0000 UTC m=+6.778515800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:12.765192 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.765057    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.765041095 +0000 UTC m=+6.778611237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:12.866385 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:12.865848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:12.866385 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.866018    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:12.866385 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.866038    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:12.866385 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.866050    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:12.866385 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:12.866102    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.866084844 +0000 UTC m=+6.879655003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:13.123100 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.122987    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:11 +0000 UTC" deadline="2028-04-27 04:07:17.254453885 +0000 UTC"
Sep 17 19:06:13.123100 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.123044    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14097h1m4.131414019s"
Sep 17 19:06:13.131453 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.131428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:13.131582 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:13.131551    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:13.132023 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.132004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:13.132132 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:13.132113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:13.132235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.132221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:13.132324 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:13.132307    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:13.856607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:13.856370    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:14.783080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:14.783141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.783236    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.783277    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.783305    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:18.783282999 +0000 UTC m=+10.796853142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:14.783371 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.783334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:18.783320542 +0000 UTC m=+10.796890683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:14.883909 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:14.883851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:14.884070 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.884018    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:14.884070 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.884042    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:14.884070 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.884054    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:14.884213 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:14.884105    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:18.884088283 +0000 UTC m=+10.897658424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:15.131677 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:15.131598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:15.131833 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:15.131723    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:15.132125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:15.132105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:15.132232 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:15.132213    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:15.132295 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:15.132264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:15.132369 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:15.132348    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:17.131431 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:17.131380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:17.131896 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:17.131472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:17.131896 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:17.131650    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:17.132043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:17.132024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:17.132165 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:17.132138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:17.132282 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:17.132249    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:18.812484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:18.812447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:18.812484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:18.812486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:18.812952 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.812607    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:18.812952 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.812633    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:18.812952 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.812680    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.812659287 +0000 UTC m=+18.826229442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:18.812952 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.812700    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.81269094 +0000 UTC m=+18.826261081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:18.913228 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:18.913191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:18.913825 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.913384    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:18.913825 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.913428    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:18.913825 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.913443    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:18.913825 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:18.913507    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:26.913487778 +0000 UTC m=+18.927057924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:19.132183 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:19.132104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:19.132332 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:19.132238    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:19.132332 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:19.132320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:19.132458 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:19.132402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:19.132529 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:19.132478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:19.132576 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:19.132547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:21.131327 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:21.131296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:21.131742 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:21.131543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:21.131742 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:21.131612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:21.131742 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:21.131717    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:21.131907 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:21.131766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:21.131907 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:21.131846    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:23.131505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:23.131465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:23.132069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:23.131510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:23.132069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:23.131510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:23.132069 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:23.131580    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:23.132069 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:23.131686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:23.132069 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:23.131764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:25.132208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:25.132124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:25.132208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:25.132167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:25.132674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:25.132124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:25.132674 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:25.132253    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:25.132674 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:25.132357    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:25.132674 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:25.132471    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:26.871848 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:26.871823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:26.871848 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:26.871855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:26.872291 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.871973    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.872291 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.871972    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.872291 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.872035    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.872017194 +0000 UTC m=+34.885587350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:26.872291 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.872049    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.8720429 +0000 UTC m=+34.885613038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:26.972821 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:26.972794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:26.972969 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.972914    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:26.972969 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.972930    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:26.972969 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.972943    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:26.973101 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:26.973001    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.972984235 +0000 UTC m=+34.986554376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:27.132255 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:27.132177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:27.132395 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:27.132295    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:27.132719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:27.132696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:27.132824 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:27.132799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:27.133020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:27.132997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:27.133137 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:27.133076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:29.131828 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:29.131800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:29.132189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:29.131800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:29.132189 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:29.131910    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:29.132189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:29.131814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:29.132189 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:29.131979    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:29.132189 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:29.132049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:30.278288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.277916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v59rs" event={"ID":"4abc53bb-f467-46c1-a12b-cc654a5e2a11","Type":"ContainerStarted","Data":"04fcbc8ca954af3cdedb719c13325dd9842bea190df304c9dee46ff6c191ccc7"}
Sep 17 19:06:30.279682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.279652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8lqz" event={"ID":"65c25afd-98a8-469e-8b20-58d793b96d3d","Type":"ContainerStarted","Data":"dc6c6287cdc82edc025799417a48fc995f4d5cb07f8acdfb4e639df7d6ddb0c7"}
Sep 17 19:06:30.281436 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.281393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal" event={"ID":"03b66c5394879f7702479f341fee142c","Type":"ContainerStarted","Data":"ea7452a76618e750b0b518ba91ecfb46e56aa785fb9b73e9c30d0f2096f174b6"}
Sep 17 19:06:30.283015 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.282983    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d60026aede2bdb637006ae8e684c1f0" containerID="76f1f578e5bf5a89a3c5238e19063dae2ff3925b14d4258ec167d19cf6d91109" exitCode=0
Sep 17 19:06:30.283135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.283072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal" event={"ID":"5d60026aede2bdb637006ae8e684c1f0","Type":"ContainerDied","Data":"76f1f578e5bf5a89a3c5238e19063dae2ff3925b14d4258ec167d19cf6d91109"}
Sep 17 19:06:30.286396 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.286376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"04434740d38bcfcc1751a58eda2aee73acd0c3c7c866ab76b268bc0f606cb505"}
Sep 17 19:06:30.286518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.286469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"023c94d40e4e75b34d0fae4fb7a0a95108c92b20b371668c28d90bfa1175fa9f"}
Sep 17 19:06:30.286580 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.286524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"8007f27992023ae27f571a55faa657677d131a9976629b23f57679b3ad06600d"}
Sep 17 19:06:30.286580 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.286573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"9df5c8689da35358093a38f8bbb3922bb09b8d41805bc2d42f68e45e4d0d6e3d"}
Sep 17 19:06:30.286681 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.286619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"7baeb0ee49a6124dd1a2aa15a6d3c7d5ca5b766fffbc405eed72281b2d90bf8e"}
Sep 17 19:06:30.288032 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.288010    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="0482846b3dc0761551e6247b137d4f18f9c6ae26eb7fb381f2495c7b00ed73a7" exitCode=0
Sep 17 19:06:30.288131 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.288085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"0482846b3dc0761551e6247b137d4f18f9c6ae26eb7fb381f2495c7b00ed73a7"}
Sep 17 19:06:30.289769 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.289744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8h59" event={"ID":"6d6199c3-283b-463c-98dc-6884c1d0ebfd","Type":"ContainerStarted","Data":"cffbdcacc46d8691af184a74e93810b4d0bba2a8a64e63cf328938459d065726"}
Sep 17 19:06:30.291529 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.291490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r29f5" event={"ID":"051aa77a-c712-4c78-bb4b-e536d73b98f1","Type":"ContainerStarted","Data":"113ce00cc85a7f744c61fb78ed2e10eb12e13fbf1f8f04c35259c13ffc804453"}
Sep 17 19:06:30.293039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.293017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qz84" event={"ID":"80ccbf37-d2de-40d3-89f8-0756439a2d4a","Type":"ContainerStarted","Data":"f3b677a4029ccc0a6159bf38c23f2a6eb31698c2b902718484b45b0e4cde2910"}
Sep 17 19:06:30.295065 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.295016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v59rs" podStartSLOduration=2.677794443 podStartE2EDuration="20.295001738s" totalImagesPullingTime="17.617207295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.294959004 +0000 UTC m=+22.308529165" watchObservedRunningTime="2026-09-17 19:06:30.295001738 +0000 UTC m=+22.308571902"
Sep 17 19:06:30.316268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.316229    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v8h59" podStartSLOduration=2.110815532 podStartE2EDuration="20.316212377s" totalImagesPullingTime="18.205396845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.315699206 +0000 UTC m=+22.329269464" watchObservedRunningTime="2026-09-17 19:06:30.316212377 +0000 UTC m=+22.329782540"
Sep 17 19:06:30.390955 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.390876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6qz84" podStartSLOduration=2.867986677 podStartE2EDuration="20.390857297s" totalImagesPullingTime="17.52287062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.390511539 +0000 UTC m=+22.404081704" watchObservedRunningTime="2026-09-17 19:06:30.390857297 +0000 UTC m=+22.404427462"
Sep 17 19:06:30.391063 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.390964    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-57.ec2.internal" podStartSLOduration=20.390958005 podStartE2EDuration="20.390958005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.369845665 +0000 UTC m=+22.383415829" watchObservedRunningTime="2026-09-17 19:06:30.390958005 +0000 UTC m=+22.404528169"
Sep 17 19:06:30.430474 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.430440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q8lqz" podStartSLOduration=2.237294788 podStartE2EDuration="20.430405375s" totalImagesPullingTime="18.193110587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.429987934 +0000 UTC m=+22.443558097" watchObservedRunningTime="2026-09-17 19:06:30.430405375 +0000 UTC m=+22.443975537"
Sep 17 19:06:30.460774 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:30.460725    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r29f5" podStartSLOduration=2.284505355 podStartE2EDuration="20.460707933s" totalImagesPullingTime="18.176202578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:30.459522663 +0000 UTC m=+22.473092824" watchObservedRunningTime="2026-09-17 19:06:30.460707933 +0000 UTC m=+22.474278095"
Sep 17 19:06:31.131813 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.131663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:31.131945 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:31.131874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:31.131945 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.131729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:31.131945 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:31.131938    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:31.132046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.131729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:31.132083 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:31.132053    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:31.297097 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.297033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mggpm" event={"ID":"96223c43-9157-4116-a57e-c191a14ae089","Type":"ContainerStarted","Data":"d04d552365b44d722166ba9d731af4dadc353c98ed661f42dca3c3bc24277da5"}
Sep 17 19:06:31.299793 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.299529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal" event={"ID":"5d60026aede2bdb637006ae8e684c1f0","Type":"ContainerStarted","Data":"996e1182d0599cae8af66c1972351c10e9f6af2f56b2c2cf8b68ef88514265c6"}
Sep 17 19:06:31.301210 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.301183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" event={"ID":"abe2d884-45cb-4b87-a658-ea74e9c8d013","Type":"ContainerStarted","Data":"b3f58f0bcfa1c4418270cb8c8ec3c40a50a685124e382d8d507618fdf59a554e"}
Sep 17 19:06:31.304958 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.304665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"1bb344725e6dbbad3c8780d558344e3b6556e3d011265bff644b2c4ca2478c7d"}
Sep 17 19:06:31.313719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.313679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mggpm" podStartSLOduration=3.744553227 podStartE2EDuration="21.313665854s" totalImagesPullingTime="17.569112627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:31.313036322 +0000 UTC m=+23.326606494" watchObservedRunningTime="2026-09-17 19:06:31.313665854 +0000 UTC m=+23.327236014"
Sep 17 19:06:31.793799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.793772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:31.794573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.794551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:31.815068 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.815028    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-57.ec2.internal" podStartSLOduration=21.815011257 podStartE2EDuration="21.815011257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:31.330340692 +0000 UTC m=+23.343910854" watchObservedRunningTime="2026-09-17 19:06:31.815011257 +0000 UTC m=+23.828581417"
Sep 17 19:06:31.852091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:31.852067    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:06:32.142024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:32.141897    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:31.852087415Z","UUID":"ced41e47-fd0a-427c-9cb8-0f69dcabb4ce","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:06:32.143497 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:32.143475    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:06:32.143620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:32.143505    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:06:32.309507 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:32.309480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" event={"ID":"abe2d884-45cb-4b87-a658-ea74e9c8d013","Type":"ContainerStarted","Data":"cdd2a4c938e40d34fa9e0a2092d632aaaa3c9f7d936d3ed71712abefaf7eb55d"}
Sep 17 19:06:33.131953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.131884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:33.132126 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.131974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:33.132126 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.131981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:33.132126 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:33.131979    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:33.132126 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:33.132057    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:33.132298 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:33.132161    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:33.313468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.313439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" event={"ID":"abe2d884-45cb-4b87-a658-ea74e9c8d013","Type":"ContainerStarted","Data":"8f7df94ef8b50c25626f3ef45cb6f04452cd56ef18a248c0af48fad5a6d8b896"}
Sep 17 19:06:33.316618 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.316601    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 19:06:33.316739 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.316624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"29ef19687cdab3cf214b962709216090ea0c4ef2ee60bf1630290f4dd2bd2b44"}
Sep 17 19:06:33.358888 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:33.358847    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bd467" podStartSLOduration=2.090607845 podStartE2EDuration="23.358829417s" totalImagesPullingTime="21.268221572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-09-17 19:06:33.357193666 +0000 UTC m=+25.370763852" watchObservedRunningTime="2026-09-17 19:06:33.358829417 +0000 UTC m=+25.372399580"
Sep 17 19:06:35.132120 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:35.132075    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:35.132566 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:35.132203    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:35.132566 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:35.132217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:35.132566 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:35.132334    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:35.132566 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:35.132206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:35.132566 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:35.132450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:36.328534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:36.328357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" event={"ID":"a332c076-59cb-4fb4-be55-3216ae0e4d2a","Type":"ContainerStarted","Data":"c4032dc141556fadd643116313546b36f79a71dd261ae1aa80e257d929f1646e"}
Sep 17 19:06:36.329073 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:36.328584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:36.330380 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:36.330357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerStarted","Data":"76588d6c793ac05245ef175a724db811ed84392623189cf830699659c385d6ef"}
Sep 17 19:06:36.343948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:36.343931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:36.356893 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:36.356850    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88" podStartSLOduration=8.190330774 podStartE2EDuration="26.356839288s" totalImagesPullingTime="18.166508514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:36.356295114 +0000 UTC m=+28.369865275" watchObservedRunningTime="2026-09-17 19:06:36.356839288 +0000 UTC m=+28.370409447"
Sep 17 19:06:37.132049 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.132005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:37.132343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.132096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:37.132343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.132215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:37.132343 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:37.132222    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:37.132343 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:37.132330    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:37.132583 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:37.132455    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:37.334675 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.334640    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="76588d6c793ac05245ef175a724db811ed84392623189cf830699659c385d6ef" exitCode=0
Sep 17 19:06:37.335078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.334718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"76588d6c793ac05245ef175a724db811ed84392623189cf830699659c385d6ef"}
Sep 17 19:06:37.335142 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.335106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:37.335183 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.335167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:37.349767 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:37.349747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:06:38.173602 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.173571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wgtr9"]
Sep 17 19:06:38.173688 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.173653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:38.173750 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:38.173733    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:38.180388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.180367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmlm6"]
Sep 17 19:06:38.180485 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.180454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:38.180543 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:38.180523    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:38.191285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.191265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gxwcj"]
Sep 17 19:06:38.191356 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.191328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:38.191409 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:38.191391    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:38.338463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.338380    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="9bdd16378c52e2082fd3e629ef2333264079d3b155c66541a226e4c3c6973ea4" exitCode=0
Sep 17 19:06:38.338463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.338445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"9bdd16378c52e2082fd3e629ef2333264079d3b155c66541a226e4c3c6973ea4"}
Sep 17 19:06:38.975675 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.975643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:38.975831 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.975794    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 19:06:38.976308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:38.976288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6qz84"
Sep 17 19:06:40.132739 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:40.132544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:40.133077 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:40.132593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:40.133077 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:40.132847    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:40.133077 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:40.132619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:40.133077 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:40.132895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:40.133077 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:40.133006    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:40.344506 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:40.344439    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="6b07f6718f16cf368877c5bc1e0c9386c7cc42e4c2bd94faadba77b0ecbb2d9e" exitCode=0
Sep 17 19:06:40.344506 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:40.344474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"6b07f6718f16cf368877c5bc1e0c9386c7cc42e4c2bd94faadba77b0ecbb2d9e"}
Sep 17 19:06:42.132038 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.132003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:42.132507 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.132003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:42.132507 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.132145    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgtr9" podUID="41df1f9f-c14e-41fb-bf22-ae8346861773"
Sep 17 19:06:42.132507 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.132161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:42.132507 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.132224    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gxwcj" podUID="8c5cff5c-f7d6-4184-90a8-7720b5051ba9"
Sep 17 19:06:42.132507 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.132298    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmlm6" podUID="e79c5ef0-ffbf-4265-9111-738e945f2e26"
Sep 17 19:06:42.895816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.895782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:42.895997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.895828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:42.895997 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.895927    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.895997 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.895964    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.895997 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.895985    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs podName:41df1f9f-c14e-41fb-bf22-ae8346861773 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.895970744 +0000 UTC m=+66.909540900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs") pod "network-metrics-daemon-wgtr9" (UID: "41df1f9f-c14e-41fb-bf22-ae8346861773") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.896225 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.896010    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret podName:8c5cff5c-f7d6-4184-90a8-7720b5051ba9 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.895997145 +0000 UTC m=+66.909567287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret") pod "global-pull-secret-syncer-gxwcj" (UID: "8c5cff5c-f7d6-4184-90a8-7720b5051ba9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.997130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:42.997104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:42.997286 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.997238    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:42.997286 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.997259    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:42.997286 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.997271    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l8n6 for pod openshift-network-diagnostics/network-check-target-vmlm6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.997397 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:42.997327    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6 podName:e79c5ef0-ffbf-4265-9111-738e945f2e26 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.997308051 +0000 UTC m=+67.010878191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l8n6" (UniqueName: "kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6") pod "network-check-target-vmlm6" (UID: "e79c5ef0-ffbf-4265-9111-738e945f2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:43.499235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.499212    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-57.ec2.internal" event="NodeReady"
Sep 17 19:06:43.499773 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.499311    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:06:43.542053 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.542018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:06:43.570267 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.570229    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-slzkf"]
Sep 17 19:06:43.570399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.570319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.581604 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.581584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:06:43.581727 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.581623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:06:43.581875 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.581860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drwjl\""
Sep 17 19:06:43.585977 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.585959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:06:43.597216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.596480    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"]
Sep 17 19:06:43.597216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.596566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.610571 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.610552    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"]
Sep 17 19:06:43.610733 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.610714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.612403 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.612381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:06:43.613171 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.613142    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.613592 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.613404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 19:06:43.624716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.624700    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.628515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.628498    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"]
Sep 17 19:06:43.628657 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.628635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"
Sep 17 19:06:43.631657 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.631639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.637589 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.637568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.637683 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.637624    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-96rrr\""
Sep 17 19:06:43.637744 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.637571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rfm7\""
Sep 17 19:06:43.637861 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.637842    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.638351 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.638333    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.638471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.638406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fvz86\""
Sep 17 19:06:43.638549 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.638533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 19:06:43.652691 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.652667    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-stjnd"]
Sep 17 19:06:43.652799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.652783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.672722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.672704    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-584d86cbf4-w6hd2"]
Sep 17 19:06:43.672828 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.672745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.678379 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.678359    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 19:06:43.694863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.694845    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 19:06:43.695010 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.694852    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 19:06:43.695244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.695225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 19:06:43.695698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.695678    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.695781 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.695750    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 19:06:43.696016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.695995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.696078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.696022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w4g8x\""
Sep 17 19:06:43.696078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.696044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 19:06:43.696620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.696603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9jgks\""
Sep 17 19:06:43.696856 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.696840    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"]
Sep 17 19:06:43.697155 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.697138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.700985 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.700963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 19:06:43.701638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.701617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.701808 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.701783    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.701872 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.701857    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9qq72\""
Sep 17 19:06:43.702021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.701708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 19:06:43.702145 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 19:06:43.702604 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.702694 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2cad30b-602b-41e5-a0d4-ff853e2973ed-serving-cert\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.702748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw42\" (UniqueName: \"kubernetes.io/projected/183b86fe-09a1-430e-9d15-e2fa1057d6f2-kube-api-access-8kw42\") pod \"volume-data-source-validator-5c4db8bdf9-49r8s\" (UID: \"183b86fe-09a1-430e-9d15-e2fa1057d6f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"
Sep 17 19:06:43.702801 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703000 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.702979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-tmp\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.703066 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703166 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rl4\" (UniqueName: \"kubernetes.io/projected/a2cad30b-602b-41e5-a0d4-ff853e2973ed-kube-api-access-q7rl4\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.703298 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e34a88e-3874-4271-a620-fc63f5c242c5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.703361 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.703434 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6tl\" (UniqueName: \"kubernetes.io/projected/5e34a88e-3874-4271-a620-fc63f5c242c5-kube-api-access-mb6tl\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.703505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-snapshots\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.703560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsgz\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703734 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.703919 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703901    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 19:06:43.703987 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.703987 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.703975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.704087 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.704006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.705295 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.704623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 19:06:43.713450 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.713409    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"]
Sep 17 19:06:43.713661 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.713619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.716165 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716139    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 19:06:43.716247 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lbl7x\""
Sep 17 19:06:43.716350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 19:06:43.716439 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 19:06:43.716498 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.716553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.716508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.730108 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.730092    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"]
Sep 17 19:06:43.730214 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.730200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.733683 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.733666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.733776 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.733751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 19:06:43.733978 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.733960    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2r4mz\""
Sep 17 19:06:43.734103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.734017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.734103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.734019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 19:06:43.750726 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:43.750726 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-slzkf"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-stjnd"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"]
Sep 17 19:06:43.750927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.750908    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"]
Sep 17 19:06:43.753666 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.753645    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.753758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.753657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7xx2h\""
Sep 17 19:06:43.753758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.753671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.753900 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.753885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 19:06:43.776821 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.776803    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xdvl6"]
Sep 17 19:06:43.776948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.776929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"
Sep 17 19:06:43.779194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.779174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.779280 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.779246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.779761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.779744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j2qhd\""
Sep 17 19:06:43.791940 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.791915    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tjfbc"]
Sep 17 19:06:43.792033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.792011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:43.794199 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.794175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:06:43.794480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.794463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:06:43.794582 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.794565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4z8hc\""
Sep 17 19:06:43.794643 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.794630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:06:43.805218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e34a88e-3874-4271-a620-fc63f5c242c5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.805321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.805321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnzrn\" (UniqueName: \"kubernetes.io/projected/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-kube-api-access-wnzrn\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.805321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-snapshots\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.805321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.805499 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsgz\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.805499 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.805357    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:43.805499 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-config\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.805499 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.805457    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.305430629 +0000 UTC m=+36.319000781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:43.805654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgnr\" (UniqueName: \"kubernetes.io/projected/f302149c-75e1-489b-8ed4-51ff1c642044-kube-api-access-mwgnr\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.805654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.805654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-trusted-ca\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.805654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.805654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-serving-cert\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.805845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2cad30b-602b-41e5-a0d4-ff853e2973ed-serving-cert\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.805845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw42\" (UniqueName: \"kubernetes.io/projected/183b86fe-09a1-430e-9d15-e2fa1057d6f2-kube-api-access-8kw42\") pod \"volume-data-source-validator-5c4db8bdf9-49r8s\" (UID: \"183b86fe-09a1-430e-9d15-e2fa1057d6f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"
Sep 17 19:06:43.805845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.805739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.806023 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.806087 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.806140 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-stats-auth\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.806140 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.806210 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5e34a88e-3874-4271-a620-fc63f5c242c5-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.806251 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6tl\" (UniqueName: \"kubernetes.io/projected/5e34a88e-3874-4271-a620-fc63f5c242c5-kube-api-access-mb6tl\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.806296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-default-certificate\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.806347 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.806347 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-snapshots\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.806447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.806520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f302149c-75e1-489b-8ed4-51ff1c642044-serving-cert\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.806666 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.806647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88t8m\" (UniqueName: \"kubernetes.io/projected/07801f2c-540e-4886-9e6b-61e74fabf220-kube-api-access-88t8m\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.807107 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.807217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.807217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.807365 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.807365 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.807328    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:43.807365 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.807344    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:43.807365 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4wm\" (UniqueName: \"kubernetes.io/projected/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-kube-api-access-kv4wm\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.807592 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.807508    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.307487461 +0000 UTC m=+36.321057602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:43.807653 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.807729 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.807845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.807902 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-config\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.807959 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.807896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.808125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"]
Sep 17 19:06:43.808191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"]
Sep 17 19:06:43.808191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"]
Sep 17 19:06:43.808191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-584d86cbf4-w6hd2"]
Sep 17 19:06:43.808337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjfbc"]
Sep 17 19:06:43.808337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:43.808337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xdvl6"]
Sep 17 19:06:43.808337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808253    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"]
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2cad30b-602b-41e5-a0d4-ff853e2973ed-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-tmp\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.809013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.809047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.808965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2cad30b-602b-41e5-a0d4-ff853e2973ed-tmp\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.809091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.809089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rl4\" (UniqueName: \"kubernetes.io/projected/a2cad30b-602b-41e5-a0d4-ff853e2973ed-kube-api-access-q7rl4\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.810876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.810857    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mr5fs\""
Sep 17 19:06:43.811153 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.811132    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:06:43.811315 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.811136    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:06:43.811814 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.811779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2cad30b-602b-41e5-a0d4-ff853e2973ed-serving-cert\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.812146 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.812124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.812388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.812341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.814807 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.814789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsgz\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.815910 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.815867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:43.816778 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.816121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6tl\" (UniqueName: \"kubernetes.io/projected/5e34a88e-3874-4271-a620-fc63f5c242c5-kube-api-access-mb6tl\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:43.816778 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.816737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw42\" (UniqueName: \"kubernetes.io/projected/183b86fe-09a1-430e-9d15-e2fa1057d6f2-kube-api-access-8kw42\") pod \"volume-data-source-validator-5c4db8bdf9-49r8s\" (UID: \"183b86fe-09a1-430e-9d15-e2fa1057d6f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"
Sep 17 19:06:43.817687 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.817667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rl4\" (UniqueName: \"kubernetes.io/projected/a2cad30b-602b-41e5-a0d4-ff853e2973ed-kube-api-access-q7rl4\") pod \"insights-operator-6f9c644dd9-slzkf\" (UID: \"a2cad30b-602b-41e5-a0d4-ff853e2973ed\") " pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.906605 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.906573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf"
Sep 17 19:06:43.909971 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.909944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-config\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.910084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.909981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.910084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzrg\" (UniqueName: \"kubernetes.io/projected/e88907d5-5333-413a-9894-be76f8494870-kube-api-access-wdzrg\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:43.910084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnzrn\" (UniqueName: \"kubernetes.io/projected/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-kube-api-access-wnzrn\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-config\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.910111    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgnr\" (UniqueName: \"kubernetes.io/projected/f302149c-75e1-489b-8ed4-51ff1c642044-kube-api-access-mwgnr\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-trusted-ca\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.910187    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.41016762 +0000 UTC m=+36.423737782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:43.910243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-serving-cert\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t9cz\" (UniqueName: \"kubernetes.io/projected/f0e461fd-9850-46cb-a766-517348080fe6-kube-api-access-5t9cz\") pod \"network-check-source-7cbc45fddc-sk75t\" (UID: \"f0e461fd-9850-46cb-a766-517348080fe6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860d54a-ff5f-4340-bca1-642b2736dc57-cert\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-stats-auth\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f282270-b758-4a96-9392-fa511f2059cb-config-volume\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:43.910572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-default-certificate\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.910603    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.410583864 +0000 UTC m=+36.424154023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f302149c-75e1-489b-8ed4-51ff1c642044-serving-cert\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88t8m\" (UniqueName: \"kubernetes.io/projected/07801f2c-540e-4886-9e6b-61e74fabf220-kube-api-access-88t8m\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-config\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4wm\" (UniqueName: \"kubernetes.io/projected/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-kube-api-access-kv4wm\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f282270-b758-4a96-9392-fa511f2059cb-tmp-dir\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2w2\" (UniqueName: \"kubernetes.io/projected/7860d54a-ff5f-4340-bca1-642b2736dc57-kube-api-access-5l2w2\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjlg\" (UniqueName: \"kubernetes.io/projected/5f282270-b758-4a96-9392-fa511f2059cb-kube-api-access-brjlg\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:43.911043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.910938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f302149c-75e1-489b-8ed4-51ff1c642044-trusted-ca\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.911884 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.911269    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:43.911884 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.911403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-config\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.911884 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:43.911785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.411767241 +0000 UTC m=+36.425337379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:43.912035 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.911933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:43.912386 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.912364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.914271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.914198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-serving-cert\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.914271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.914201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.914271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.914270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f302149c-75e1-489b-8ed4-51ff1c642044-serving-cert\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.914506 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.914443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-stats-auth\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.914549 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.914496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-default-certificate\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.921111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.921087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnzrn\" (UniqueName: \"kubernetes.io/projected/b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975-kube-api-access-wnzrn\") pod \"service-ca-operator-57cdc4b768-q272f\" (UID: \"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:43.921111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.921105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgnr\" (UniqueName: \"kubernetes.io/projected/f302149c-75e1-489b-8ed4-51ff1c642044-kube-api-access-mwgnr\") pod \"console-operator-fd4bb46c-stjnd\" (UID: \"f302149c-75e1-489b-8ed4-51ff1c642044\") " pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:43.921274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.921213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4wm\" (UniqueName: \"kubernetes.io/projected/c2b2f1a3-3474-4579-9259-8db6e55ddbf8-kube-api-access-kv4wm\") pod \"kube-storage-version-migrator-operator-55889f7b47-hfq2j\" (UID: \"c2b2f1a3-3474-4579-9259-8db6e55ddbf8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:43.923065 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.923045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88t8m\" (UniqueName: \"kubernetes.io/projected/07801f2c-540e-4886-9e6b-61e74fabf220-kube-api-access-88t8m\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:43.938788 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.938770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"
Sep 17 19:06:43.982206 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:43.982186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:44.011438 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:44.011438 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t9cz\" (UniqueName: \"kubernetes.io/projected/f0e461fd-9850-46cb-a766-517348080fe6-kube-api-access-5t9cz\") pod \"network-check-source-7cbc45fddc-sk75t\" (UID: \"f0e461fd-9850-46cb-a766-517348080fe6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"
Sep 17 19:06:44.011573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860d54a-ff5f-4340-bca1-642b2736dc57-cert\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:44.011573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f282270-b758-4a96-9392-fa511f2059cb-config-volume\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.011573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.011573 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.011515    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 19:06:44.011573 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.011565    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls podName:e88907d5-5333-413a-9894-be76f8494870 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.51154985 +0000 UTC m=+36.525120009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-6sbg8" (UID: "e88907d5-5333-413a-9894-be76f8494870") : secret "samples-operator-tls" not found
Sep 17 19:06:44.011854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f282270-b758-4a96-9392-fa511f2059cb-tmp-dir\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.011854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2w2\" (UniqueName: \"kubernetes.io/projected/7860d54a-ff5f-4340-bca1-642b2736dc57-kube-api-access-5l2w2\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:44.011854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjlg\" (UniqueName: \"kubernetes.io/projected/5f282270-b758-4a96-9392-fa511f2059cb-kube-api-access-brjlg\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.011854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.011800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzrg\" (UniqueName: \"kubernetes.io/projected/e88907d5-5333-413a-9894-be76f8494870-kube-api-access-wdzrg\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:44.012050 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.012012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f282270-b758-4a96-9392-fa511f2059cb-tmp-dir\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.012104 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.012083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f282270-b758-4a96-9392-fa511f2059cb-config-volume\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.012170 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.012087    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:44.012205 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.012188    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.512170904 +0000 UTC m=+36.525741044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:44.014217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.014196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7860d54a-ff5f-4340-bca1-642b2736dc57-cert\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:44.023707 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.023686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzrg\" (UniqueName: \"kubernetes.io/projected/e88907d5-5333-413a-9894-be76f8494870-kube-api-access-wdzrg\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:44.023798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.023719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t9cz\" (UniqueName: \"kubernetes.io/projected/f0e461fd-9850-46cb-a766-517348080fe6-kube-api-access-5t9cz\") pod \"network-check-source-7cbc45fddc-sk75t\" (UID: \"f0e461fd-9850-46cb-a766-517348080fe6\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"
Sep 17 19:06:44.023860 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.023840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2w2\" (UniqueName: \"kubernetes.io/projected/7860d54a-ff5f-4340-bca1-642b2736dc57-kube-api-access-5l2w2\") pod \"ingress-canary-xdvl6\" (UID: \"7860d54a-ff5f-4340-bca1-642b2736dc57\") " pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:44.023953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.023932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjlg\" (UniqueName: \"kubernetes.io/projected/5f282270-b758-4a96-9392-fa511f2059cb-kube-api-access-brjlg\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.024724 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.024709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"
Sep 17 19:06:44.039801 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.039780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"
Sep 17 19:06:44.097141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.097121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"
Sep 17 19:06:44.101305 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.101285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xdvl6"
Sep 17 19:06:44.135367 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.135349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:06:44.135490 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.135473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:06:44.135546 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.135504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:06:44.137861 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.137843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:06:44.138118 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.138104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:06:44.138314 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.138295    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jlzv\""
Sep 17 19:06:44.138477 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.138462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2b6xk\""
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.314457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.314519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.314561    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.314629    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.314610406 +0000 UTC m=+37.328180565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.314667    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:44.314687 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.314685    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:44.315059 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.314739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.314725425 +0000 UTC m=+37.328295565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:44.415255 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.415223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:44.415450 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.415275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:44.415450 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.415387    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:44.415450 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.415410    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.415390125 +0000 UTC m=+37.428960289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:44.415609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.415510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:44.415609 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.415530    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.415509139 +0000 UTC m=+37.429079282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:44.415609 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.415590    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:44.415751 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.415630    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.415617076 +0000 UTC m=+37.429187230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:44.516158 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.516128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:44.516612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:44.516212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:44.516612 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.516267    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 19:06:44.516612 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.516323    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:44.516612 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.516347    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls podName:e88907d5-5333-413a-9894-be76f8494870 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.516330055 +0000 UTC m=+37.529900214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-6sbg8" (UID: "e88907d5-5333-413a-9894-be76f8494870") : secret "samples-operator-tls" not found
Sep 17 19:06:44.516612 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:44.516375    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.516362845 +0000 UTC m=+37.529933003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:45.325501 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.325341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:45.325501 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.325496    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:45.325621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.325526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:45.325621 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.325549    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.32553442 +0000 UTC m=+39.339104563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:45.325621 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.325590    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:45.325621 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.325600    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:45.325769 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.325658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.325647629 +0000 UTC m=+39.339217768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:45.426069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.426045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:45.426170 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.426154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:45.426230 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.426191    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:45.426230 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.426203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:45.426331 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.426247    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.426228231 +0000 UTC m=+39.439798384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:45.426331 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.426275    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:45.426331 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.426306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.426294848 +0000 UTC m=+39.439864988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:45.426459 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.426387    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.426375864 +0000 UTC m=+39.439946005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:45.471831 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.471797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xdvl6"]
Sep 17 19:06:45.471947 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.471908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s"]
Sep 17 19:06:45.476645 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.476610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-stjnd"]
Sep 17 19:06:45.478235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.478210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-slzkf"]
Sep 17 19:06:45.481723 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.481465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t"]
Sep 17 19:06:45.481723 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.481513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j"]
Sep 17 19:06:45.481824 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.481783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f"]
Sep 17 19:06:45.482975 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:45.482944    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b2f1a3_3474_4579_9259_8db6e55ddbf8.slice/crio-5f747f15535ab0e2c49131d89f5035d107a5613bf7b98a518a9598cf73c1c5c8 WatchSource:0}: Error finding container 5f747f15535ab0e2c49131d89f5035d107a5613bf7b98a518a9598cf73c1c5c8: Status 404 returned error can't find the container with id 5f747f15535ab0e2c49131d89f5035d107a5613bf7b98a518a9598cf73c1c5c8
Sep 17 19:06:45.484261 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:45.484238    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf302149c_75e1_489b_8ed4_51ff1c642044.slice/crio-bd3d3486770bdfaa2d6db0f8746a08dfe9d5900d78488d29b3a0b6b35b1cccc6 WatchSource:0}: Error finding container bd3d3486770bdfaa2d6db0f8746a08dfe9d5900d78488d29b3a0b6b35b1cccc6: Status 404 returned error can't find the container with id bd3d3486770bdfaa2d6db0f8746a08dfe9d5900d78488d29b3a0b6b35b1cccc6
Sep 17 19:06:45.489264 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:45.489194    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e461fd_9850_46cb_a766_517348080fe6.slice/crio-21ee56ed6e62dcfb869502354c7bb45e3031e4a48b666b9e6cfdc65f665c0089 WatchSource:0}: Error finding container 21ee56ed6e62dcfb869502354c7bb45e3031e4a48b666b9e6cfdc65f665c0089: Status 404 returned error can't find the container with id 21ee56ed6e62dcfb869502354c7bb45e3031e4a48b666b9e6cfdc65f665c0089
Sep 17 19:06:45.527163 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.527140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:45.527863 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.527280    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:45.527863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:45.527295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:45.527863 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.527334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.527315688 +0000 UTC m=+39.540885829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:45.527863 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.527370    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 19:06:45.527863 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:45.527409    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls podName:e88907d5-5333-413a-9894-be76f8494870 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:47.527397456 +0000 UTC m=+39.540967613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-6sbg8" (UID: "e88907d5-5333-413a-9894-be76f8494870") : secret "samples-operator-tls" not found
Sep 17 19:06:46.296078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.296033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:46.369211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.369119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xdvl6" event={"ID":"7860d54a-ff5f-4340-bca1-642b2736dc57","Type":"ContainerStarted","Data":"b7e90bf5a6aabd31c1d00336c036c3f37ea14327daba2ebaf439bc8c8886f2f4"}
Sep 17 19:06:46.371941 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.371857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s" event={"ID":"183b86fe-09a1-430e-9d15-e2fa1057d6f2","Type":"ContainerStarted","Data":"31e5b513ecde04958942400deddf779b04b10ddb90662f10f304bb94b45ba83f"}
Sep 17 19:06:46.378971 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.378898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf" event={"ID":"a2cad30b-602b-41e5-a0d4-ff853e2973ed","Type":"ContainerStarted","Data":"1f3ddaef2d4e0a42e28747801d1bf5c80f731fa2ed321ed940130b3bf9060a08"}
Sep 17 19:06:46.383337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.383255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" event={"ID":"f302149c-75e1-489b-8ed4-51ff1c642044","Type":"ContainerStarted","Data":"bd3d3486770bdfaa2d6db0f8746a08dfe9d5900d78488d29b3a0b6b35b1cccc6"}
Sep 17 19:06:46.392024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.391940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t" event={"ID":"f0e461fd-9850-46cb-a766-517348080fe6","Type":"ContainerStarted","Data":"21ee56ed6e62dcfb869502354c7bb45e3031e4a48b666b9e6cfdc65f665c0089"}
Sep 17 19:06:46.395879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.395836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:46.400157 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.399305    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="94a2c09b7d6adaf303d0332a3947f8e26039a314491726758cbe144cac85464f" exitCode=0
Sep 17 19:06:46.400157 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.399379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"94a2c09b7d6adaf303d0332a3947f8e26039a314491726758cbe144cac85464f"}
Sep 17 19:06:46.405170 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.404780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f" event={"ID":"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975","Type":"ContainerStarted","Data":"9a78454ab34dcbed618842c9c485c0b30335cbe4e5e0ced33f5aa76d1679f785"}
Sep 17 19:06:46.407385 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.407280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j" event={"ID":"c2b2f1a3-3474-4579-9259-8db6e55ddbf8","Type":"ContainerStarted","Data":"5f747f15535ab0e2c49131d89f5035d107a5613bf7b98a518a9598cf73c1c5c8"}
Sep 17 19:06:46.464668 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:46.464616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.344923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.345170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.345346    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.345361    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.345442    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.345403116 +0000 UTC m=+43.358973272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.345621    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:47.345700 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.345671    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.34565668 +0000 UTC m=+43.359226821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:47.422843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.422554    2581 generic.go:356] "Generic (PLEG): container finished" podID="152784a6-5c9f-4a87-8a4f-873c218f9f37" containerID="e065d65b530d3add37b3c0a3f7e7fa669a2e4c8ce3839d737eed34229e011722" exitCode=0
Sep 17 19:06:47.422843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.422613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerDied","Data":"e065d65b530d3add37b3c0a3f7e7fa669a2e4c8ce3839d737eed34229e011722"}
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.445983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.446044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.446135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.446237    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.446295    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.446277438 +0000 UTC m=+43.459847581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.446522    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:47.446597 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.446572    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.446557069 +0000 UTC m=+43.460127218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:47.447026 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.446646    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.446635176 +0000 UTC m=+43.460205337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.546931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.547021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.547407    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.547454    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.547487    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls podName:e88907d5-5333-413a-9894-be76f8494870 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.547467542 +0000 UTC m=+43.561037687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-6sbg8" (UID: "e88907d5-5333-413a-9894-be76f8494870") : secret "samples-operator-tls" not found
Sep 17 19:06:47.548318 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:47.547986    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.547966849 +0000 UTC m=+43.561536991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:47.705739 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.705603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:47.781479 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:47.781353    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:48.429869 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:48.429797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" event={"ID":"152784a6-5c9f-4a87-8a4f-873c218f9f37","Type":"ContainerStarted","Data":"38475fa24c24f12e6c5ea89e7ad892b40f70e779ee8650afa80a8cd949049357"}
Sep 17 19:06:48.452996 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:48.452951    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ptw2z" podStartSLOduration=6.594695897 podStartE2EDuration="38.452935416s" totalImagesPullingTime="31.858239519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:48.451129955 +0000 UTC m=+40.464700117" watchObservedRunningTime="2026-09-17 19:06:48.452935416 +0000 UTC m=+40.466505634"
Sep 17 19:06:48.581075 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:48.581043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:48.681628 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:48.681551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.435045 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:49.435005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s" event={"ID":"183b86fe-09a1-430e-9d15-e2fa1057d6f2","Type":"ContainerStarted","Data":"fa0c81e026f51b214cfc742815cca9ab49b07d9eab3eb450d6341eca101b5fe9"}
Sep 17 19:06:49.457599 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:49.457554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-49r8s" podStartSLOduration=17.296944168 podStartE2EDuration="19.457538413s" totalImagesPullingTime="2.160594245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:49.456842771 +0000 UTC m=+41.470412933" watchObservedRunningTime="2026-09-17 19:06:49.457538413 +0000 UTC m=+41.471108573"
Sep 17 19:06:49.608162 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:49.608139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.679992 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:49.679966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.385384 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.385345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:51.385384 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.385388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:51.385768 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.385512    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:51.385768 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.385542    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:51.385768 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.385553    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:51.385768 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.385586    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.385567172 +0000 UTC m=+51.399137316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:51.385768 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.385605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.385597099 +0000 UTC m=+51.399167243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:51.485791 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.485750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:51.485962 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.485881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:51.485962 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.485911    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:51.485962 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.485927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:51.486124 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.485977    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.485957128 +0000 UTC m=+51.499527286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:51.486124 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.486040    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.486022989 +0000 UTC m=+51.499593130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:51.486124 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.486061    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:51.486124 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.486122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.486108874 +0000 UTC m=+51.499679017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:51.586780 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.586748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:51.586931 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:51.586799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:51.586931 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.586908    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:51.587034 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.586958    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.586944039 +0000 UTC m=+51.600514182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:51.587083 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.587041    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 19:06:51.587114 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:51.587105    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls podName:e88907d5-5333-413a-9894-be76f8494870 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:59.587088522 +0000 UTC m=+51.600658671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-6sbg8" (UID: "e88907d5-5333-413a-9894-be76f8494870") : secret "samples-operator-tls" not found
Sep 17 19:06:53.451665 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.451624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf" event={"ID":"a2cad30b-602b-41e5-a0d4-ff853e2973ed","Type":"ContainerStarted","Data":"e723192647fe858c8bbb7dd7b9a3358e5b127a185ef884ae4acaa5f82c555e08"}
Sep 17 19:06:53.453761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.453737    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/0.log"
Sep 17 19:06:53.453885 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.453780    2581 generic.go:356] "Generic (PLEG): container finished" podID="f302149c-75e1-489b-8ed4-51ff1c642044" containerID="d278afa1ce9ac150b71f7c62a7cc51ba03885724997625d0e3841d7df8ea0e78" exitCode=255
Sep 17 19:06:53.453885 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.453866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" event={"ID":"f302149c-75e1-489b-8ed4-51ff1c642044","Type":"ContainerDied","Data":"d278afa1ce9ac150b71f7c62a7cc51ba03885724997625d0e3841d7df8ea0e78"}
Sep 17 19:06:53.454067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.454050    2581 scope.go:117] "RemoveContainer" containerID="d278afa1ce9ac150b71f7c62a7cc51ba03885724997625d0e3841d7df8ea0e78"
Sep 17 19:06:53.455972 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.455627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t" event={"ID":"f0e461fd-9850-46cb-a766-517348080fe6","Type":"ContainerStarted","Data":"fb1b31549073bce603b210f62e9bb1f3abc231f9dccf59db9eb02986eca67333"}
Sep 17 19:06:53.457560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.457517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f" event={"ID":"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975","Type":"ContainerStarted","Data":"91c89784486bdb31b77bb4b6670c6c4c6a0d9f0e65f597a6c5b8b5fa03dd09d0"}
Sep 17 19:06:53.459362 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.459337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j" event={"ID":"c2b2f1a3-3474-4579-9259-8db6e55ddbf8","Type":"ContainerStarted","Data":"49d54770afcbab6712cc8104be4e8bfceecc3c7bc3ba4eb40df7d23b0a9d09c1"}
Sep 17 19:06:53.461327 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.461306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xdvl6" event={"ID":"7860d54a-ff5f-4340-bca1-642b2736dc57","Type":"ContainerStarted","Data":"45bc051de409a0077f8aa3deae2cba932e7d5cc27b72789970eaa0a4f3728129"}
Sep 17 19:06:53.553593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.553546    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf" podStartSLOduration=16.145662437 podStartE2EDuration="23.553527907s" totalImagesPullingTime="7.40786547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.55005497 +0000 UTC m=+45.563625132" watchObservedRunningTime="2026-09-17 19:06:53.553527907 +0000 UTC m=+45.567098070"
Sep 17 19:06:53.660574 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.660491    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-sk75t" podStartSLOduration=15.240138351 podStartE2EDuration="22.660470992s" totalImagesPullingTime="7.420332641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.621712182 +0000 UTC m=+45.635282344" watchObservedRunningTime="2026-09-17 19:06:53.660470992 +0000 UTC m=+45.674041146"
Sep 17 19:06:53.660704 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.660609    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xdvl6" podStartSLOduration=3.253368772 podStartE2EDuration="10.66060106s" totalImagesPullingTime="7.407232288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.660153614 +0000 UTC m=+45.673723774" watchObservedRunningTime="2026-09-17 19:06:53.66060106 +0000 UTC m=+45.674171221"
Sep 17 19:06:53.722976 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.722734    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f" podStartSLOduration=15.314202419 podStartE2EDuration="22.722714431s" totalImagesPullingTime="7.408512012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.721914292 +0000 UTC m=+45.735484455" watchObservedRunningTime="2026-09-17 19:06:53.722714431 +0000 UTC m=+45.736284594"
Sep 17 19:06:53.744021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.743998    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.846026 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.845980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.937523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.937498    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.982782 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.982731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:53.982782 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:53.982782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:06:54.200707 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.200646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j" podStartSLOduration=15.788260368 podStartE2EDuration="23.200623281s" totalImagesPullingTime="7.412362913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.74864866 +0000 UTC m=+45.762218821" watchObservedRunningTime="2026-09-17 19:06:54.200623281 +0000 UTC m=+46.214193443"
Sep 17 19:06:54.200859 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.200806    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5jzgh"]
Sep 17 19:06:54.218768 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.218742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jzgh"]
Sep 17 19:06:54.218914 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.218865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.221623 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.221601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:06:54.221733 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.221692    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:06:54.221802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.221766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ftw2s\""
Sep 17 19:06:54.253978 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.253921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:54.312177 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.312152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9968eda-be9c-48e1-b82f-9f3519b726cb-crio-socket\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.312297 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.312222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.312339 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.312304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6c8\" (UniqueName: \"kubernetes.io/projected/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-api-access-fj6c8\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.312377 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.312347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9968eda-be9c-48e1-b82f-9f3519b726cb-data-volume\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.312457 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.312435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.413157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6c8\" (UniqueName: \"kubernetes.io/projected/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-api-access-fj6c8\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.413206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9968eda-be9c-48e1-b82f-9f3519b726cb-data-volume\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.413243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.413351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9968eda-be9c-48e1-b82f-9f3519b726cb-crio-socket\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.413452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.413567    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.413624    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.913606002 +0000 UTC m=+46.927176161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.414310    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.414353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9968eda-be9c-48e1-b82f-9f3519b726cb-data-volume\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.414370    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.91435364 +0000 UTC m=+46.927923794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:54.414395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.414367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9968eda-be9c-48e1-b82f-9f3519b726cb-crio-socket\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.430054 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.430036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6c8\" (UniqueName: \"kubernetes.io/projected/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-api-access-fj6c8\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.466550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.466525    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/1.log"
Sep 17 19:06:54.467035 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.467016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/0.log"
Sep 17 19:06:54.467087 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.467064    2581 generic.go:356] "Generic (PLEG): container finished" podID="f302149c-75e1-489b-8ed4-51ff1c642044" containerID="f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139" exitCode=255
Sep 17 19:06:54.467168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.467143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" event={"ID":"f302149c-75e1-489b-8ed4-51ff1c642044","Type":"ContainerDied","Data":"f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139"}
Sep 17 19:06:54.467231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.467202    2581 scope.go:117] "RemoveContainer" containerID="d278afa1ce9ac150b71f7c62a7cc51ba03885724997625d0e3841d7df8ea0e78"
Sep 17 19:06:54.467460 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.467442    2581 scope.go:117] "RemoveContainer" containerID="f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139"
Sep 17 19:06:54.467644 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.467622    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-stjnd_openshift-console-operator(f302149c-75e1-489b-8ed4-51ff1c642044)\"" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podUID="f302149c-75e1-489b-8ed4-51ff1c642044"
Sep 17 19:06:54.845211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.845182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"]
Sep 17 19:06:54.876099 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.876081    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"]
Sep 17 19:06:54.876236 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.876219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"
Sep 17 19:06:54.879410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.879393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:54.879514 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.879393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 19:06:54.879912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.879892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rn7ln\""
Sep 17 19:06:54.917194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.917166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvn4p\" (UniqueName: \"kubernetes.io/projected/2bc1564e-c179-4312-a165-8d68df6075e5-kube-api-access-xvn4p\") pod \"migrator-76794cbd6c-9vn7w\" (UID: \"2bc1564e-c179-4312-a165-8d68df6075e5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"
Sep 17 19:06:54.917288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.917222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.917288 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.917285    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:54.917364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:54.917295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:54.917364 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.917324    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:55.9173127 +0000 UTC m=+47.930882838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:54.917473 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.917369    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:54.917473 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:54.917444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:55.917409201 +0000 UTC m=+47.930979341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:55.018555 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.018535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvn4p\" (UniqueName: \"kubernetes.io/projected/2bc1564e-c179-4312-a165-8d68df6075e5-kube-api-access-xvn4p\") pod \"migrator-76794cbd6c-9vn7w\" (UID: \"2bc1564e-c179-4312-a165-8d68df6075e5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"
Sep 17 19:06:55.027885 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.027867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvn4p\" (UniqueName: \"kubernetes.io/projected/2bc1564e-c179-4312-a165-8d68df6075e5-kube-api-access-xvn4p\") pod \"migrator-76794cbd6c-9vn7w\" (UID: \"2bc1564e-c179-4312-a165-8d68df6075e5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"
Sep 17 19:06:55.188954 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.188937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"
Sep 17 19:06:55.366117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.366095    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r29f5_051aa77a-c712-4c78-bb4b-e536d73b98f1/dns-node-resolver/0.log"
Sep 17 19:06:55.473842 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.473771    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/1.log"
Sep 17 19:06:55.474166 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.474147    2581 scope.go:117] "RemoveContainer" containerID="f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139"
Sep 17 19:06:55.474314 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:55.474296    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-stjnd_openshift-console-operator(f302149c-75e1-489b-8ed4-51ff1c642044)\"" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podUID="f302149c-75e1-489b-8ed4-51ff1c642044"
Sep 17 19:06:55.480652 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.480491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w"]
Sep 17 19:06:55.482833 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:55.482807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc1564e_c179_4312_a165_8d68df6075e5.slice/crio-68c8e08dd63366bd9ff028a6849df2429ec86f6e1a9704ee840bd0aea945fc55 WatchSource:0}: Error finding container 68c8e08dd63366bd9ff028a6849df2429ec86f6e1a9704ee840bd0aea945fc55: Status 404 returned error can't find the container with id 68c8e08dd63366bd9ff028a6849df2429ec86f6e1a9704ee840bd0aea945fc55
Sep 17 19:06:55.926619 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.926593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:55.926726 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:55.926649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:55.926726 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:55.926706    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:55.926800 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:55.926769    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.92675315 +0000 UTC m=+49.940323292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:55.926800 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:55.926772    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:55.926874 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:55.926808    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.926797745 +0000 UTC m=+49.940367884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:56.365773 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.365712    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v59rs_4abc53bb-f467-46c1-a12b-cc654a5e2a11/node-ca/0.log"
Sep 17 19:06:56.478683 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.478648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w" event={"ID":"2bc1564e-c179-4312-a165-8d68df6075e5","Type":"ContainerStarted","Data":"68c8e08dd63366bd9ff028a6849df2429ec86f6e1a9704ee840bd0aea945fc55"}
Sep 17 19:06:56.870518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.870487    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pqfrv"]
Sep 17 19:06:56.887624 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.887605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:56.890559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.890541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pqfrv"]
Sep 17 19:06:56.891119 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.891097    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 19:06:56.891219 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.891159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 19:06:56.891975 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.891956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x47vw\""
Sep 17 19:06:56.891975 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.891964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 19:06:56.892134 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.892006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 19:06:56.934917 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.934896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-cabundle\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:56.934995 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.934949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-key\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:56.935033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.935005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmkf\" (UniqueName: \"kubernetes.io/projected/3ab03222-2725-425e-9d9a-b0e7b8c24981-kube-api-access-bzmkf\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:56.964337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:56.964314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xdvl6_7860d54a-ff5f-4340-bca1-642b2736dc57/serve-healthcheck-canary/0.log"
Sep 17 19:06:57.035934 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.035913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-cabundle\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.036011 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.035961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-key\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.036011 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.035997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmkf\" (UniqueName: \"kubernetes.io/projected/3ab03222-2725-425e-9d9a-b0e7b8c24981-kube-api-access-bzmkf\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.036686 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.036670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-cabundle\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.038400 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.038383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab03222-2725-425e-9d9a-b0e7b8c24981-signing-key\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.044680 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.044661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmkf\" (UniqueName: \"kubernetes.io/projected/3ab03222-2725-425e-9d9a-b0e7b8c24981-kube-api-access-bzmkf\") pod \"service-ca-55f6fc7b86-pqfrv\" (UID: \"3ab03222-2725-425e-9d9a-b0e7b8c24981\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.197920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.197903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv"
Sep 17 19:06:57.484224 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.484194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w" event={"ID":"2bc1564e-c179-4312-a165-8d68df6075e5","Type":"ContainerStarted","Data":"85b4d51bbaf8d299c148f4355b5e27eeca463379f592a2575ec73b35dc0f9f61"}
Sep 17 19:06:57.484538 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.484245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w" event={"ID":"2bc1564e-c179-4312-a165-8d68df6075e5","Type":"ContainerStarted","Data":"3ae33a22f326bf14d2a9c98dc625f49995ea9030c03996ab61c3cb38e2d351f3"}
Sep 17 19:06:57.490593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.490568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pqfrv"]
Sep 17 19:06:57.493245 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:06:57.493220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab03222_2725_425e_9d9a_b0e7b8c24981.slice/crio-82e45f775c584fedfc05b456b0c32d56bd17c42088fa57e92ce7d44909990326 WatchSource:0}: Error finding container 82e45f775c584fedfc05b456b0c32d56bd17c42088fa57e92ce7d44909990326: Status 404 returned error can't find the container with id 82e45f775c584fedfc05b456b0c32d56bd17c42088fa57e92ce7d44909990326
Sep 17 19:06:57.943240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.943210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:57.943383 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:57.943352    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:57.943469 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:57.943376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:06:57.943469 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:57.943404    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:01.943388713 +0000 UTC m=+53.956958852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:57.943580 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:57.943468    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:57.943580 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:57.943527    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:01.943511099 +0000 UTC m=+53.957081253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:58.488477 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:58.488441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv" event={"ID":"3ab03222-2725-425e-9d9a-b0e7b8c24981","Type":"ContainerStarted","Data":"6a598e7525e07a326e2c794d697b5c6dfc4f82c3537a17d090195eb49b7c2446"}
Sep 17 19:06:58.488844 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:58.488500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv" event={"ID":"3ab03222-2725-425e-9d9a-b0e7b8c24981","Type":"ContainerStarted","Data":"82e45f775c584fedfc05b456b0c32d56bd17c42088fa57e92ce7d44909990326"}
Sep 17 19:06:58.513408 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:58.513360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-pqfrv" podStartSLOduration=2.513345374 podStartE2EDuration="2.513345374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.512945733 +0000 UTC m=+50.526515896" watchObservedRunningTime="2026-09-17 19:06:58.513345374 +0000 UTC m=+50.526915534"
Sep 17 19:06:58.513929 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:58.513899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9vn7w" podStartSLOduration=2.906497311 podStartE2EDuration="4.513889287s" totalImagesPullingTime="1.607391976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:57.511908342 +0000 UTC m=+49.525478503" watchObservedRunningTime="2026-09-17 19:06:58.513889287 +0000 UTC m=+50.527459452"
Sep 17 19:06:59.455684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.455648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:06:59.455854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.455713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:06:59.455854 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.455798    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:59.455854 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.455813    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:06:59.455854 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.455830    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bfb59776-v9qqd: secret "image-registry-tls" not found
Sep 17 19:06:59.455995 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.455866    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls podName:5e34a88e-3874-4271-a620-fc63f5c242c5 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.455849363 +0000 UTC m=+67.469419502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-7pxn9" (UID: "5e34a88e-3874-4271-a620-fc63f5c242c5") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:59.455995 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.455889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls podName:100146e7-01ae-4192-b7c4-e72231811447 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.455872192 +0000 UTC m=+67.469442333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls") pod "image-registry-64bfb59776-v9qqd" (UID: "100146e7-01ae-4192-b7c4-e72231811447") : secret "image-registry-tls" not found
Sep 17 19:06:59.557029 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.557001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.557125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.557134    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.557172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.557193    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.557174205 +0000 UTC m=+67.570744349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : secret "router-metrics-certs-default" not found
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.557266    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.557314    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert podName:53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.557301481 +0000 UTC m=+67.570871644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7qrhw" (UID: "53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9") : secret "networking-console-plugin-cert" not found
Sep 17 19:06:59.557381 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.557342    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle podName:07801f2c-540e-4886-9e6b-61e74fabf220 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.557322301 +0000 UTC m=+67.570892456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle") pod "router-default-584d86cbf4-w6hd2" (UID: "07801f2c-540e-4886-9e6b-61e74fabf220") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:59.658308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.658267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:59.658499 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.658342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:06:59.658557 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.658515    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:59.658601 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:06:59.658589    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls podName:5f282270-b758-4a96-9392-fa511f2059cb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.658569376 +0000 UTC m=+67.672139532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls") pod "dns-default-tjfbc" (UID: "5f282270-b758-4a96-9392-fa511f2059cb") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:59.660792 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.660762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e88907d5-5333-413a-9894-be76f8494870-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-6sbg8\" (UID: \"e88907d5-5333-413a-9894-be76f8494870\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:06:59.959946 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:06:59.959920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"
Sep 17 19:07:00.252855 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:00.252789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8"]
Sep 17 19:07:00.496761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:00.496737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8" event={"ID":"e88907d5-5333-413a-9894-be76f8494870","Type":"ContainerStarted","Data":"a0aac0c897787fc04c513e7bccbad0fc7e99b678f4b86e2e71f3e5bec7c16d2e"}
Sep 17 19:07:01.979523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:01.979490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:01.979937 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:01.979551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:01.979937 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:01.979647    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:01.979937 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:01.979699    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls podName:e9968eda-be9c-48e1-b82f-9f3519b726cb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:09.979685698 +0000 UTC m=+61.993255837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jzgh" (UID: "e9968eda-be9c-48e1-b82f-9f3519b726cb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:01.980106 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:01.979948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9968eda-be9c-48e1-b82f-9f3519b726cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:02.504990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:02.504957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8" event={"ID":"e88907d5-5333-413a-9894-be76f8494870","Type":"ContainerStarted","Data":"f717f26c5fcafe0d73d3ecbeeb2e0905ed1aaba171d71472dc316ab5b0d04f9c"}
Sep 17 19:07:02.505111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:02.505010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8" event={"ID":"e88907d5-5333-413a-9894-be76f8494870","Type":"ContainerStarted","Data":"bb514262e378cfedfdba9141a51200d35036541c4a323d20eca882957b36f090"}
Sep 17 19:07:02.524038 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:02.523995    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-6sbg8" podStartSLOduration=29.813135618 podStartE2EDuration="31.523980617s" totalImagesPullingTime="1.710844999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:02.523381987 +0000 UTC m=+54.536952149" watchObservedRunningTime="2026-09-17 19:07:02.523980617 +0000 UTC m=+54.537550778"
Sep 17 19:07:03.982780 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:03.982751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:03.983127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:03.982803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:03.983127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:03.983119    2581 scope.go:117] "RemoveContainer" containerID="f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139"
Sep 17 19:07:04.514434 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.514390    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:07:04.514812 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.514799    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/1.log"
Sep 17 19:07:04.514871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.514835    2581 generic.go:356] "Generic (PLEG): container finished" podID="f302149c-75e1-489b-8ed4-51ff1c642044" containerID="4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b" exitCode=255
Sep 17 19:07:04.514871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.514866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" event={"ID":"f302149c-75e1-489b-8ed4-51ff1c642044","Type":"ContainerDied","Data":"4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b"}
Sep 17 19:07:04.514934 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.514917    2581 scope.go:117] "RemoveContainer" containerID="f62445245a2c6f252cbb6ea85f3724ab0a8b2890dd103b94720aedb396278139"
Sep 17 19:07:04.515272 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:04.515250    2581 scope.go:117] "RemoveContainer" containerID="4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b"
Sep 17 19:07:04.515491 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:04.515473    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-stjnd_openshift-console-operator(f302149c-75e1-489b-8ed4-51ff1c642044)\"" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podUID="f302149c-75e1-489b-8ed4-51ff1c642044"
Sep 17 19:07:05.519519 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:05.519494    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:07:09.359851 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:09.359819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bl88"
Sep 17 19:07:10.042701 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.042669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:10.045237 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.045211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9968eda-be9c-48e1-b82f-9f3519b726cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jzgh\" (UID: \"e9968eda-be9c-48e1-b82f-9f3519b726cb\") " pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:10.130743 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.130710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ftw2s\""
Sep 17 19:07:10.138063 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.138043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jzgh"
Sep 17 19:07:10.455501 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.455385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jzgh"]
Sep 17 19:07:10.457582 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:10.457555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9968eda_be9c_48e1_b82f_9f3519b726cb.slice/crio-95ee60f8a63ed92b17a13e3016fca2fdf1073599b7d23508133daf9761d64b02 WatchSource:0}: Error finding container 95ee60f8a63ed92b17a13e3016fca2fdf1073599b7d23508133daf9761d64b02: Status 404 returned error can't find the container with id 95ee60f8a63ed92b17a13e3016fca2fdf1073599b7d23508133daf9761d64b02
Sep 17 19:07:10.538709 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.538684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jzgh" event={"ID":"e9968eda-be9c-48e1-b82f-9f3519b726cb","Type":"ContainerStarted","Data":"0409a3623f33162315025079f5209cb4a20f2a777b5efe5d5840706063c04b5c"}
Sep 17 19:07:10.538842 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:10.538734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jzgh" event={"ID":"e9968eda-be9c-48e1-b82f-9f3519b726cb","Type":"ContainerStarted","Data":"95ee60f8a63ed92b17a13e3016fca2fdf1073599b7d23508133daf9761d64b02"}
Sep 17 19:07:11.545401 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:11.545367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jzgh" event={"ID":"e9968eda-be9c-48e1-b82f-9f3519b726cb","Type":"ContainerStarted","Data":"4e4484ad3bbefcc59951774ce7b160cfaa4dd026e6a9ddc3f312111ae38ace4e"}
Sep 17 19:07:12.551746 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:12.551710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jzgh" event={"ID":"e9968eda-be9c-48e1-b82f-9f3519b726cb","Type":"ContainerStarted","Data":"4ff5e29a0e0191bb1bbc217246b918d7ca8a9d041316a919563c09c7db4e0aed"}
Sep 17 19:07:13.982932 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:13.982897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:13.983307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:13.982985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:13.983307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:13.983226    2581 scope.go:117] "RemoveContainer" containerID="4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b"
Sep 17 19:07:13.983387 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:13.983367    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-stjnd_openshift-console-operator(f302149c-75e1-489b-8ed4-51ff1c642044)\"" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podUID="f302149c-75e1-489b-8ed4-51ff1c642044"
Sep 17 19:07:14.002339 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.002182    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5jzgh" podStartSLOduration=18.265741108 podStartE2EDuration="20.002165906s" totalImagesPullingTime="1.736424798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:12.589152888 +0000 UTC m=+64.602723049" watchObservedRunningTime="2026-09-17 19:07:14.002165906 +0000 UTC m=+66.015736067"
Sep 17 19:07:14.559774 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.559751    2581 scope.go:117] "RemoveContainer" containerID="4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b"
Sep 17 19:07:14.559909 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:14.559889    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-stjnd_openshift-console-operator(f302149c-75e1-489b-8ed4-51ff1c642044)\"" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podUID="f302149c-75e1-489b-8ed4-51ff1c642044"
Sep 17 19:07:14.981449 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.981398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:07:14.981449 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.981459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:07:14.985170 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.985150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:14.985494 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.985203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:14.994686 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.994663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c5cff5c-f7d6-4184-90a8-7720b5051ba9-original-pull-secret\") pod \"global-pull-secret-syncer-gxwcj\" (UID: \"8c5cff5c-f7d6-4184-90a8-7720b5051ba9\") " pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:07:14.994787 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:14.994747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41df1f9f-c14e-41fb-bf22-ae8346861773-metrics-certs\") pod \"network-metrics-daemon-wgtr9\" (UID: \"41df1f9f-c14e-41fb-bf22-ae8346861773\") " pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:07:15.048202 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.048176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gxwcj"
Sep 17 19:07:15.064058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.064035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jlzv\""
Sep 17 19:07:15.072220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.072201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgtr9"
Sep 17 19:07:15.081972 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.081864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:07:15.084405 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.084382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8n6\" (UniqueName: \"kubernetes.io/projected/e79c5ef0-ffbf-4265-9111-738e945f2e26-kube-api-access-2l8n6\") pod \"network-check-target-vmlm6\" (UID: \"e79c5ef0-ffbf-4265-9111-738e945f2e26\") " pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:07:15.326025 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.325999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gxwcj"]
Sep 17 19:07:15.328225 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:15.328197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5cff5c_f7d6_4184_90a8_7720b5051ba9.slice/crio-c9da1c645ede79266b2dec3360b2ecea8b1d572e5b05ca65fcbb20070ab06577 WatchSource:0}: Error finding container c9da1c645ede79266b2dec3360b2ecea8b1d572e5b05ca65fcbb20070ab06577: Status 404 returned error can't find the container with id c9da1c645ede79266b2dec3360b2ecea8b1d572e5b05ca65fcbb20070ab06577
Sep 17 19:07:15.350973 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.350953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wgtr9"]
Sep 17 19:07:15.354709 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:15.354684    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41df1f9f_c14e_41fb_bf22_ae8346861773.slice/crio-aeae1e2536c2f8041787b21eeafef12c770a9d8816b1470ae96331354c2f8c28 WatchSource:0}: Error finding container aeae1e2536c2f8041787b21eeafef12c770a9d8816b1470ae96331354c2f8c28: Status 404 returned error can't find the container with id aeae1e2536c2f8041787b21eeafef12c770a9d8816b1470ae96331354c2f8c28
Sep 17 19:07:15.356894 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.356877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2b6xk\""
Sep 17 19:07:15.365310 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.365289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:07:15.486044 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.486010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:07:15.486173 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.486068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:07:15.488474 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.488447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"image-registry-64bfb59776-v9qqd\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") " pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:07:15.488474 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.488468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e34a88e-3874-4271-a620-fc63f5c242c5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-7pxn9\" (UID: \"5e34a88e-3874-4271-a620-fc63f5c242c5\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:07:15.564284 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.564254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gxwcj" event={"ID":"8c5cff5c-f7d6-4184-90a8-7720b5051ba9","Type":"ContainerStarted","Data":"c9da1c645ede79266b2dec3360b2ecea8b1d572e5b05ca65fcbb20070ab06577"}
Sep 17 19:07:15.565390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.565368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgtr9" event={"ID":"41df1f9f-c14e-41fb-bf22-ae8346861773","Type":"ContainerStarted","Data":"aeae1e2536c2f8041787b21eeafef12c770a9d8816b1470ae96331354c2f8c28"}
Sep 17 19:07:15.586747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.586692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:15.586816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.586749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:07:15.586816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.586807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:15.587451 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.587403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07801f2c-540e-4886-9e6b-61e74fabf220-service-ca-bundle\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:15.589005 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.588983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7qrhw\" (UID: \"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:07:15.589081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.589059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07801f2c-540e-4886-9e6b-61e74fabf220-metrics-certs\") pod \"router-default-584d86cbf4-w6hd2\" (UID: \"07801f2c-540e-4886-9e6b-61e74fabf220\") " pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:15.636383 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.636364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmlm6"]
Sep 17 19:07:15.638720 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:15.638696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79c5ef0_ffbf_4265_9111_738e945f2e26.slice/crio-5be0415b7f130e756556da26085ec3a414ae21b0215218bcbfbd290a6b7026ec WatchSource:0}: Error finding container 5be0415b7f130e756556da26085ec3a414ae21b0215218bcbfbd290a6b7026ec: Status 404 returned error can't find the container with id 5be0415b7f130e756556da26085ec3a414ae21b0215218bcbfbd290a6b7026ec
Sep 17 19:07:15.684945 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.684930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drwjl\""
Sep 17 19:07:15.687631 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.687614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:07:15.689989 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.689972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f282270-b758-4a96-9392-fa511f2059cb-metrics-tls\") pod \"dns-default-tjfbc\" (UID: \"5f282270-b758-4a96-9392-fa511f2059cb\") " pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:07:15.691931 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.691918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:07:15.723490 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.723462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rfm7\""
Sep 17 19:07:15.730965 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.730943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"
Sep 17 19:07:15.767440 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.766622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w4g8x\""
Sep 17 19:07:15.773987 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.773901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"
Sep 17 19:07:15.818514 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.818483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9qq72\""
Sep 17 19:07:15.826663 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.826632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:15.939669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.939643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mr5fs\""
Sep 17 19:07:15.947176 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.947148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:07:15.971191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:15.971141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:07:15.974664 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:15.974634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100146e7_01ae_4192_b7c4_e72231811447.slice/crio-6a54839a2709c94fa4216cc8a92d858a3f9997041440d298b94c26da82136d1e WatchSource:0}: Error finding container 6a54839a2709c94fa4216cc8a92d858a3f9997041440d298b94c26da82136d1e: Status 404 returned error can't find the container with id 6a54839a2709c94fa4216cc8a92d858a3f9997041440d298b94c26da82136d1e
Sep 17 19:07:16.015478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.015179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9"]
Sep 17 19:07:16.021981 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:16.021527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e34a88e_3874_4271_a620_fc63f5c242c5.slice/crio-adf1550dcf01ab5b78d3a96d2dbd6d9d5c2958e00a30812498db4149f5d97cc2 WatchSource:0}: Error finding container adf1550dcf01ab5b78d3a96d2dbd6d9d5c2958e00a30812498db4149f5d97cc2: Status 404 returned error can't find the container with id adf1550dcf01ab5b78d3a96d2dbd6d9d5c2958e00a30812498db4149f5d97cc2
Sep 17 19:07:16.071453 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.070976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw"]
Sep 17 19:07:16.075371 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:16.074095    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fb5fd1_2275_4cf0_bd83_3f3f49b12ee9.slice/crio-130b1cfe8ad6c3328c240cd752090daabd6fe8ecbfcbd1fa0665ec4e468aeaaa WatchSource:0}: Error finding container 130b1cfe8ad6c3328c240cd752090daabd6fe8ecbfcbd1fa0665ec4e468aeaaa: Status 404 returned error can't find the container with id 130b1cfe8ad6c3328c240cd752090daabd6fe8ecbfcbd1fa0665ec4e468aeaaa
Sep 17 19:07:16.121628 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.121602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-584d86cbf4-w6hd2"]
Sep 17 19:07:16.125503 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:16.125480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07801f2c_540e_4886_9e6b_61e74fabf220.slice/crio-8e4e649da3fb159e67d0aceda0b63f1cbb1a4e89500824bece47ec707e59954c WatchSource:0}: Error finding container 8e4e649da3fb159e67d0aceda0b63f1cbb1a4e89500824bece47ec707e59954c: Status 404 returned error can't find the container with id 8e4e649da3fb159e67d0aceda0b63f1cbb1a4e89500824bece47ec707e59954c
Sep 17 19:07:16.243354 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.243321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjfbc"]
Sep 17 19:07:16.525539 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:16.525501    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f282270_b758_4a96_9392_fa511f2059cb.slice/crio-a0bed3e320e1be4fc0696ec9db1c17a4c554670d4ee45488a95addb2125e3527 WatchSource:0}: Error finding container a0bed3e320e1be4fc0696ec9db1c17a4c554670d4ee45488a95addb2125e3527: Status 404 returned error can't find the container with id a0bed3e320e1be4fc0696ec9db1c17a4c554670d4ee45488a95addb2125e3527
Sep 17 19:07:16.572228 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.572187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-584d86cbf4-w6hd2" event={"ID":"07801f2c-540e-4886-9e6b-61e74fabf220","Type":"ContainerStarted","Data":"d5b93a19d2e6144a3c792d274fe437a94e0821f688eef5f864f48c0f154b6ddf"}
Sep 17 19:07:16.572376 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.572246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-584d86cbf4-w6hd2" event={"ID":"07801f2c-540e-4886-9e6b-61e74fabf220","Type":"ContainerStarted","Data":"8e4e649da3fb159e67d0aceda0b63f1cbb1a4e89500824bece47ec707e59954c"}
Sep 17 19:07:16.574064 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.574037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" event={"ID":"100146e7-01ae-4192-b7c4-e72231811447","Type":"ContainerStarted","Data":"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"}
Sep 17 19:07:16.574194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.574089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:07:16.574194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.574109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" event={"ID":"100146e7-01ae-4192-b7c4-e72231811447","Type":"ContainerStarted","Data":"6a54839a2709c94fa4216cc8a92d858a3f9997041440d298b94c26da82136d1e"}
Sep 17 19:07:16.575786 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.575757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmlm6" event={"ID":"e79c5ef0-ffbf-4265-9111-738e945f2e26","Type":"ContainerStarted","Data":"e8ea62d79b62a00323445f541e8d5d2eb801ba98308852ec240eaea43bd02f0f"}
Sep 17 19:07:16.575899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.575810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmlm6" event={"ID":"e79c5ef0-ffbf-4265-9111-738e945f2e26","Type":"ContainerStarted","Data":"5be0415b7f130e756556da26085ec3a414ae21b0215218bcbfbd290a6b7026ec"}
Sep 17 19:07:16.575899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.575850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:07:16.578559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.578394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9" event={"ID":"5e34a88e-3874-4271-a620-fc63f5c242c5","Type":"ContainerStarted","Data":"adf1550dcf01ab5b78d3a96d2dbd6d9d5c2958e00a30812498db4149f5d97cc2"}
Sep 17 19:07:16.579844 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.579823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw" event={"ID":"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9","Type":"ContainerStarted","Data":"130b1cfe8ad6c3328c240cd752090daabd6fe8ecbfcbd1fa0665ec4e468aeaaa"}
Sep 17 19:07:16.581919 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.581873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjfbc" event={"ID":"5f282270-b758-4a96-9392-fa511f2059cb","Type":"ContainerStarted","Data":"a0bed3e320e1be4fc0696ec9db1c17a4c554670d4ee45488a95addb2125e3527"}
Sep 17 19:07:16.593860 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.593810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-584d86cbf4-w6hd2" podStartSLOduration=46.593797698 podStartE2EDuration="46.593797698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:16.592654098 +0000 UTC m=+68.606224282" watchObservedRunningTime="2026-09-17 19:07:16.593797698 +0000 UTC m=+68.607367886"
Sep 17 19:07:16.632829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.632755    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vmlm6" podStartSLOduration=66.632742495 podStartE2EDuration="1m6.632742495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:16.611507399 +0000 UTC m=+68.625077564" watchObservedRunningTime="2026-09-17 19:07:16.632742495 +0000 UTC m=+68.646312646"
Sep 17 19:07:16.633187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.633157    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" podStartSLOduration=66.633148427 podStartE2EDuration="1m6.633148427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:16.632087818 +0000 UTC m=+68.645658001" watchObservedRunningTime="2026-09-17 19:07:16.633148427 +0000 UTC m=+68.646718587"
Sep 17 19:07:16.827943 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.827854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:16.830811 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:16.830593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:17.592769 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:17.592734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgtr9" event={"ID":"41df1f9f-c14e-41fb-bf22-ae8346861773","Type":"ContainerStarted","Data":"c387b9bc79dafee6619ca054763f2e635189eb902469e3ebc7d5e54245fa07da"}
Sep 17 19:07:17.593221 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:17.592803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgtr9" event={"ID":"41df1f9f-c14e-41fb-bf22-ae8346861773","Type":"ContainerStarted","Data":"234ee4284514a524a946f05d9c241d8508bf8755899d42dde5b40a40b0176a69"}
Sep 17 19:07:17.594695 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:17.594672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:17.598963 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:17.598446    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-584d86cbf4-w6hd2"
Sep 17 19:07:17.619089 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:17.619035    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wgtr9" podStartSLOduration=66.355184658 podStartE2EDuration="1m7.619015341s" totalImagesPullingTime="1.263830683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:17.615152112 +0000 UTC m=+69.628722274" watchObservedRunningTime="2026-09-17 19:07:17.619015341 +0000 UTC m=+69.632585527"
Sep 17 19:07:18.458667 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.458624    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"]
Sep 17 19:07:18.475277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.475239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"]
Sep 17 19:07:18.475449 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.475405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.478578 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.478527    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 19:07:18.479132 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.479111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 19:07:18.479523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.479507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 19:07:18.479611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.479514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-r2dq8\""
Sep 17 19:07:18.480162 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.480137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 19:07:18.495808 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.495789    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"]
Sep 17 19:07:18.495937 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.495921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.498296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.498276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 19:07:18.509405 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.509386    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"]
Sep 17 19:07:18.509514 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.509439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"]
Sep 17 19:07:18.509514 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.509461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"]
Sep 17 19:07:18.509582 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.509523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.512370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.512349    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 19:07:18.512479 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.512461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 19:07:18.512552 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.512470    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 19:07:18.512552 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.512504    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 19:07:18.612562 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzjx\" (UniqueName: \"kubernetes.io/projected/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-kube-api-access-2nzjx\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246e5ec4-13a7-44f6-8b76-a84819f61738-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.612970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.612826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkn2\" (UniqueName: \"kubernetes.io/projected/209bd5a0-beda-4d79-9527-10b9a45cb4cd-kube-api-access-2pkn2\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.613388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.613356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.613520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.613398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/209bd5a0-beda-4d79-9527-10b9a45cb4cd-tmp\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.613520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.613443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/209bd5a0-beda-4d79-9527-10b9a45cb4cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.613520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.613493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqm6g\" (UniqueName: \"kubernetes.io/projected/246e5ec4-13a7-44f6-8b76-a84819f61738-kube-api-access-sqm6g\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.714694 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.714629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/209bd5a0-beda-4d79-9527-10b9a45cb4cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.714694 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.714668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqm6g\" (UniqueName: \"kubernetes.io/projected/246e5ec4-13a7-44f6-8b76-a84819f61738-kube-api-access-sqm6g\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.714889 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.714743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715034 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715162 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715162 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzjx\" (UniqueName: \"kubernetes.io/projected/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-kube-api-access-2nzjx\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246e5ec4-13a7-44f6-8b76-a84819f61738-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.715273 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkn2\" (UniqueName: \"kubernetes.io/projected/209bd5a0-beda-4d79-9527-10b9a45cb4cd-kube-api-access-2pkn2\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.715447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.715447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/209bd5a0-beda-4d79-9527-10b9a45cb4cd-tmp\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.716081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.715830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/209bd5a0-beda-4d79-9527-10b9a45cb4cd-tmp\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.716081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.716054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.718054 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.718017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-ca\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.718146 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.718027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.719003 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.718981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.719644 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.719600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/209bd5a0-beda-4d79-9527-10b9a45cb4cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.720072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.720053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246e5ec4-13a7-44f6-8b76-a84819f61738-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.720168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.720111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-hub\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.739129 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.739086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzjx\" (UniqueName: \"kubernetes.io/projected/0b68e8cd-f347-41b3-a9e2-d6e30746da0c-kube-api-access-2nzjx\") pod \"cluster-proxy-proxy-agent-64dcd5fb57-w9m5r\" (UID: \"0b68e8cd-f347-41b3-a9e2-d6e30746da0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:18.745067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.745048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkn2\" (UniqueName: \"kubernetes.io/projected/209bd5a0-beda-4d79-9527-10b9a45cb4cd-kube-api-access-2pkn2\") pod \"klusterlet-addon-workmgr-5b9b475bcc-q42bl\" (UID: \"209bd5a0-beda-4d79-9527-10b9a45cb4cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.748955 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.748931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqm6g\" (UniqueName: \"kubernetes.io/projected/246e5ec4-13a7-44f6-8b76-a84819f61738-kube-api-access-sqm6g\") pod \"managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt\" (UID: \"246e5ec4-13a7-44f6-8b76-a84819f61738\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.794990 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.794965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"
Sep 17 19:07:18.806130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.806106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:18.818427 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:18.818389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"
Sep 17 19:07:21.359456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.358980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r"]
Sep 17 19:07:21.360256 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:21.360214    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b68e8cd_f347_41b3_a9e2_d6e30746da0c.slice/crio-2d43a321c47b5a8cc4ef7772613d365ae6f867ad60132c91acc5183bed97a6eb WatchSource:0}: Error finding container 2d43a321c47b5a8cc4ef7772613d365ae6f867ad60132c91acc5183bed97a6eb: Status 404 returned error can't find the container with id 2d43a321c47b5a8cc4ef7772613d365ae6f867ad60132c91acc5183bed97a6eb
Sep 17 19:07:21.360837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.360758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"]
Sep 17 19:07:21.365361 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:21.365330    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209bd5a0_beda_4d79_9527_10b9a45cb4cd.slice/crio-ac00b043e293326c1d4f43606c711e84286c1790ba156240cb678c3fe8a0342b WatchSource:0}: Error finding container ac00b043e293326c1d4f43606c711e84286c1790ba156240cb678c3fe8a0342b: Status 404 returned error can't find the container with id ac00b043e293326c1d4f43606c711e84286c1790ba156240cb678c3fe8a0342b
Sep 17 19:07:21.371592 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.371569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt"]
Sep 17 19:07:21.613811 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.613729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gxwcj" event={"ID":"8c5cff5c-f7d6-4184-90a8-7720b5051ba9","Type":"ContainerStarted","Data":"2e540a3b1c9c5391d6e21c4f3be2613e4f4d5cecb978cbbb61bdcc1e5590711b"}
Sep 17 19:07:21.614956 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.614922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r" event={"ID":"0b68e8cd-f347-41b3-a9e2-d6e30746da0c","Type":"ContainerStarted","Data":"2d43a321c47b5a8cc4ef7772613d365ae6f867ad60132c91acc5183bed97a6eb"}
Sep 17 19:07:21.616324 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.616298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt" event={"ID":"246e5ec4-13a7-44f6-8b76-a84819f61738","Type":"ContainerStarted","Data":"ecab4b2b72141811d1969d09104582c4d31d7049a80559597474c3d73dc5c30d"}
Sep 17 19:07:21.617735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.617704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl" event={"ID":"209bd5a0-beda-4d79-9527-10b9a45cb4cd","Type":"ContainerStarted","Data":"ac00b043e293326c1d4f43606c711e84286c1790ba156240cb678c3fe8a0342b"}
Sep 17 19:07:21.619523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.619502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9" event={"ID":"5e34a88e-3874-4271-a620-fc63f5c242c5","Type":"ContainerStarted","Data":"6b4d09515dbf4be318631d4e16fd383e7cdc721daf9040dceef4c75ef12cca20"}
Sep 17 19:07:21.621018 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.620992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw" event={"ID":"53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9","Type":"ContainerStarted","Data":"9aef9c7b55de07fa3e031ccee8be770e540d809f410c589afe500e4e9e05bf86"}
Sep 17 19:07:21.622833 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.622812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjfbc" event={"ID":"5f282270-b758-4a96-9392-fa511f2059cb","Type":"ContainerStarted","Data":"73db016d6c921e8ff67d0f00eed3c0567db90b84d3a650434ea604a0c1242b76"}
Sep 17 19:07:21.622930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.622856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjfbc" event={"ID":"5f282270-b758-4a96-9392-fa511f2059cb","Type":"ContainerStarted","Data":"8f6370ad6fb3d811acba3625537f49d45856ebdf29db09d514af7dab9675cf80"}
Sep 17 19:07:21.622930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.622891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:07:21.632816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.632777    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gxwcj" podStartSLOduration=65.878308359 podStartE2EDuration="1m11.632763595s" totalImagesPullingTime="5.754455236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:21.632236141 +0000 UTC m=+73.645806304" watchObservedRunningTime="2026-09-17 19:07:21.632763595 +0000 UTC m=+73.646333755"
Sep 17 19:07:21.688185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.688000    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tjfbc" podStartSLOduration=34.146729453 podStartE2EDuration="38.687982233s" totalImagesPullingTime="4.54125278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:21.687263876 +0000 UTC m=+73.700834038" watchObservedRunningTime="2026-09-17 19:07:21.687982233 +0000 UTC m=+73.701552396"
Sep 17 19:07:21.689279 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.689242    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7qrhw" podStartSLOduration=44.697548189 podStartE2EDuration="49.689233441s" totalImagesPullingTime="4.991685252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:21.658519478 +0000 UTC m=+73.672089663" watchObservedRunningTime="2026-09-17 19:07:21.689233441 +0000 UTC m=+73.702803605"
Sep 17 19:07:21.720185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.720136    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-7pxn9" podStartSLOduration=46.675894559 podStartE2EDuration="51.720119008s" totalImagesPullingTime="5.044224449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:21.719121771 +0000 UTC m=+73.732691934" watchObservedRunningTime="2026-09-17 19:07:21.720119008 +0000 UTC m=+73.733689178"
Sep 17 19:07:21.766932 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.766905    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"]
Sep 17 19:07:21.787967 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.787946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"]
Sep 17 19:07:21.788076 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.788046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:21.790828 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.790798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 19:07:21.790828 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.790806    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f8c5z\""
Sep 17 19:07:21.842665 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.842640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-z2txv\" (UID: \"ec453613-ef16-4c60-9f0c-25dbb16eec5b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:21.943958 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:21.943932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-z2txv\" (UID: \"ec453613-ef16-4c60-9f0c-25dbb16eec5b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:21.944100 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:21.944077    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 19:07:21.944193 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:07:21.944183    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates podName:ec453613-ef16-4c60-9f0c-25dbb16eec5b nodeName:}" failed. No retries permitted until 2026-09-17 19:07:22.444159599 +0000 UTC m=+74.457729760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-z2txv" (UID: "ec453613-ef16-4c60-9f0c-25dbb16eec5b") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 19:07:22.447686 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:22.447621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-z2txv\" (UID: \"ec453613-ef16-4c60-9f0c-25dbb16eec5b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:22.455908 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:22.455795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec453613-ef16-4c60-9f0c-25dbb16eec5b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-z2txv\" (UID: \"ec453613-ef16-4c60-9f0c-25dbb16eec5b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:22.699568 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:22.699441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:23.053467 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:23.053340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"]
Sep 17 19:07:23.054673 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:23.054631    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec453613_ef16_4c60_9f0c_25dbb16eec5b.slice/crio-8c148c25d3db0c554f670b588fd8fd972941a231a2e3a9b7a62d653562ee1ab6 WatchSource:0}: Error finding container 8c148c25d3db0c554f670b588fd8fd972941a231a2e3a9b7a62d653562ee1ab6: Status 404 returned error can't find the container with id 8c148c25d3db0c554f670b588fd8fd972941a231a2e3a9b7a62d653562ee1ab6
Sep 17 19:07:23.644495 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:23.644455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv" event={"ID":"ec453613-ef16-4c60-9f0c-25dbb16eec5b","Type":"ContainerStarted","Data":"8c148c25d3db0c554f670b588fd8fd972941a231a2e3a9b7a62d653562ee1ab6"}
Sep 17 19:07:28.669352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.669263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r" event={"ID":"0b68e8cd-f347-41b3-a9e2-d6e30746da0c","Type":"ContainerStarted","Data":"6812202f9c8db057f3b5eb25cf2357b7b489748ab0a0dd77b481a02ba4b003c3"}
Sep 17 19:07:28.669352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.669315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r" event={"ID":"0b68e8cd-f347-41b3-a9e2-d6e30746da0c","Type":"ContainerStarted","Data":"46382570c90a8bb406bcd27dd3b6fea67bf4295dfaa07f5033d896a2ccea8020"}
Sep 17 19:07:28.669352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.669328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r" event={"ID":"0b68e8cd-f347-41b3-a9e2-d6e30746da0c","Type":"ContainerStarted","Data":"b58037ae31fb3e68d2c34d27b8979d68075de1f3ed6f41468a561c32b0bfe197"}
Sep 17 19:07:28.670866 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.670831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt" event={"ID":"246e5ec4-13a7-44f6-8b76-a84819f61738","Type":"ContainerStarted","Data":"de8827da9f47aea68b72178cc9493a2f924ea14b2aa73a565fca9b656b09bc90"}
Sep 17 19:07:28.672286 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.672255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl" event={"ID":"209bd5a0-beda-4d79-9527-10b9a45cb4cd","Type":"ContainerStarted","Data":"4bb5922e440df8738d57a4f62ca759551c8fddb23d6f35a60d2040a44483eebb"}
Sep 17 19:07:28.672402 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.672348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:28.674025 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.674000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv" event={"ID":"ec453613-ef16-4c60-9f0c-25dbb16eec5b","Type":"ContainerStarted","Data":"ee5affb06a734dda3aaa082c5c8dc60c5202a808c57febce2bf3922041b09293"}
Sep 17 19:07:28.674201 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.674183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:28.674392 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.674375    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl"
Sep 17 19:07:28.678394 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.678375    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv"
Sep 17 19:07:28.711351 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.711308    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64dcd5fb57-w9m5r" podStartSLOduration=3.861368207 podStartE2EDuration="10.711294747s" totalImagesPullingTime="6.84992654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.710126183 +0000 UTC m=+80.723696344" watchObservedRunningTime="2026-09-17 19:07:28.711294747 +0000 UTC m=+80.724864942"
Sep 17 19:07:28.729014 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.728972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fc4b9c74-kjhzt" podStartSLOduration=3.898488581 podStartE2EDuration="10.728959637s" totalImagesPullingTime="6.830471056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.727812833 +0000 UTC m=+80.741382991" watchObservedRunningTime="2026-09-17 19:07:28.728959637 +0000 UTC m=+80.742529797"
Sep 17 19:07:28.748534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.748500    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-z2txv" podStartSLOduration=2.5607137140000003 podStartE2EDuration="7.748489317s" totalImagesPullingTime="5.187775603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.747708108 +0000 UTC m=+80.761278268" watchObservedRunningTime="2026-09-17 19:07:28.748489317 +0000 UTC m=+80.762059477"
Sep 17 19:07:28.771829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:28.771792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b9b475bcc-q42bl" podStartSLOduration=3.926625884 podStartE2EDuration="10.771780977s" totalImagesPullingTime="6.845155093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.771403651 +0000 UTC m=+80.784973829" watchObservedRunningTime="2026-09-17 19:07:28.771780977 +0000 UTC m=+80.785351138"
Sep 17 19:07:30.133861 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:30.133827    2581 scope.go:117] "RemoveContainer" containerID="4b909126fa7d7ba0b9d533d0a5e658d18f8d50022e96c058ee7bf3064db6216b"
Sep 17 19:07:30.684700 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:30.684675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:07:30.684838 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:30.684734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" event={"ID":"f302149c-75e1-489b-8ed4-51ff1c642044","Type":"ContainerStarted","Data":"1153b8ba7696a8a99e8f0751905b5e874487006840a2bc44c89c77c8947b18c1"}
Sep 17 19:07:30.709177 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:30.709139    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd" podStartSLOduration=52.298987956 podStartE2EDuration="59.709128047s" totalImagesPullingTime="7.410140091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.708844713 +0000 UTC m=+82.722414876" watchObservedRunningTime="2026-09-17 19:07:30.709128047 +0000 UTC m=+82.722698217"
Sep 17 19:07:31.639364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:31.639329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tjfbc"
Sep 17 19:07:34.562404 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.562369    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kl6s8"]
Sep 17 19:07:34.567903 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.567875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.573543 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.573523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:34.574455 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.574434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:34.574735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.574704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wssgm\""
Sep 17 19:07:34.575323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.574742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:34.577600 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.577576    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:34.648104 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-root\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648210 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648210 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-tls\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-wtmp\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clfd\" (UniqueName: \"kubernetes.io/projected/71e6adf5-5412-4d50-96e8-bc32c04ce607-kube-api-access-4clfd\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648496 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-sys\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648496 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-metrics-client-ca\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.648599 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.648574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-textfile\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749448 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-root\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-tls\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-root\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-wtmp\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clfd\" (UniqueName: \"kubernetes.io/projected/71e6adf5-5412-4d50-96e8-bc32c04ce607-kube-api-access-4clfd\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-sys\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-metrics-client-ca\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-sys\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.749796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-wtmp\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.750031 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.749800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-textfile\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.750112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.750083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-accelerators-collector-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.750236 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.750136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-textfile\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.750236 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.750173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e6adf5-5412-4d50-96e8-bc32c04ce607-metrics-client-ca\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.752096 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.752077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-tls\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.752283 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.752268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e6adf5-5412-4d50-96e8-bc32c04ce607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.762216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.762183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clfd\" (UniqueName: \"kubernetes.io/projected/71e6adf5-5412-4d50-96e8-bc32c04ce607-kube-api-access-4clfd\") pod \"node-exporter-kl6s8\" (UID: \"71e6adf5-5412-4d50-96e8-bc32c04ce607\") " pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.877808 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:34.877744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kl6s8"
Sep 17 19:07:34.891715 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:07:34.891692    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e6adf5_5412_4d50_96e8_bc32c04ce607.slice/crio-d33fe95f63d376744eda0835f0a96f921e3db77b086156ef446869ff77100f05 WatchSource:0}: Error finding container d33fe95f63d376744eda0835f0a96f921e3db77b086156ef446869ff77100f05: Status 404 returned error can't find the container with id d33fe95f63d376744eda0835f0a96f921e3db77b086156ef446869ff77100f05
Sep 17 19:07:35.698319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:35.698266    2581 patch_prober.go:28] interesting pod/image-registry-64bfb59776-v9qqd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 19:07:35.698806 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:35.698338    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" podUID="100146e7-01ae-4192-b7c4-e72231811447" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:07:35.710231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:35.710198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl6s8" event={"ID":"71e6adf5-5412-4d50-96e8-bc32c04ce607","Type":"ContainerStarted","Data":"d33fe95f63d376744eda0835f0a96f921e3db77b086156ef446869ff77100f05"}
Sep 17 19:07:36.716497 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:36.716465    2581 generic.go:356] "Generic (PLEG): container finished" podID="71e6adf5-5412-4d50-96e8-bc32c04ce607" containerID="b8cb468769456f25ce41b44b62ba940d7b358c9d54d8532435efbd7a8d02c922" exitCode=0
Sep 17 19:07:36.716911 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:36.716526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl6s8" event={"ID":"71e6adf5-5412-4d50-96e8-bc32c04ce607","Type":"ContainerDied","Data":"b8cb468769456f25ce41b44b62ba940d7b358c9d54d8532435efbd7a8d02c922"}
Sep 17 19:07:37.598912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:37.598886    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:07:37.723841 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:37.723812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl6s8" event={"ID":"71e6adf5-5412-4d50-96e8-bc32c04ce607","Type":"ContainerStarted","Data":"c7eecdae02e72de1f1fcf817f5bc08e4d19884552a81d0426e1b53ea744b2260"}
Sep 17 19:07:37.724183 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:37.723856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kl6s8" event={"ID":"71e6adf5-5412-4d50-96e8-bc32c04ce607","Type":"ContainerStarted","Data":"5e062434033d142820c79ef093159e075bf36546b045608972ed3e6f206a106b"}
Sep 17 19:07:37.744771 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:37.744706    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kl6s8" podStartSLOduration=2.731498707 podStartE2EDuration="3.744687121s" totalImagesPullingTime="942.891715ms" totalInitContainerRuntime="70.296699ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:37.743783166 +0000 UTC m=+89.757353326" watchObservedRunningTime="2026-09-17 19:07:37.744687121 +0000 UTC m=+89.758257281"
Sep 17 19:07:40.685597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:40.685559    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:40.689354 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:40.689326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-stjnd"
Sep 17 19:07:47.525606 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:47.525571    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:07:47.596300 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:47.596273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vmlm6"
Sep 17 19:07:58.806135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.806103    2581 generic.go:356] "Generic (PLEG): container finished" podID="a2cad30b-602b-41e5-a0d4-ff853e2973ed" containerID="e723192647fe858c8bbb7dd7b9a3358e5b127a185ef884ae4acaa5f82c555e08" exitCode=0
Sep 17 19:07:58.806607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.806180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf" event={"ID":"a2cad30b-602b-41e5-a0d4-ff853e2973ed","Type":"ContainerDied","Data":"e723192647fe858c8bbb7dd7b9a3358e5b127a185ef884ae4acaa5f82c555e08"}
Sep 17 19:07:58.806607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.806544    2581 scope.go:117] "RemoveContainer" containerID="e723192647fe858c8bbb7dd7b9a3358e5b127a185ef884ae4acaa5f82c555e08"
Sep 17 19:07:58.807814 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.807793    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2b2f1a3-3474-4579-9259-8db6e55ddbf8" containerID="49d54770afcbab6712cc8104be4e8bfceecc3c7bc3ba4eb40df7d23b0a9d09c1" exitCode=0
Sep 17 19:07:58.807897 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.807835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j" event={"ID":"c2b2f1a3-3474-4579-9259-8db6e55ddbf8","Type":"ContainerDied","Data":"49d54770afcbab6712cc8104be4e8bfceecc3c7bc3ba4eb40df7d23b0a9d09c1"}
Sep 17 19:07:58.808072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:58.808060    2581 scope.go:117] "RemoveContainer" containerID="49d54770afcbab6712cc8104be4e8bfceecc3c7bc3ba4eb40df7d23b0a9d09c1"
Sep 17 19:07:59.816570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:59.816529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-slzkf" event={"ID":"a2cad30b-602b-41e5-a0d4-ff853e2973ed","Type":"ContainerStarted","Data":"ff804943589dd681bd0eabbe8999a8c9674613907c55c5f4915595c11618fc88"}
Sep 17 19:07:59.818698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:07:59.818676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-hfq2j" event={"ID":"c2b2f1a3-3474-4579-9259-8db6e55ddbf8","Type":"ContainerStarted","Data":"140a8db20acc73491693d7e13cce814e176606028420ee041f186f45c976d8e7"}
Sep 17 19:08:12.545375 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.545307    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" podUID="100146e7-01ae-4192-b7c4-e72231811447" containerName="registry" containerID="cri-o://6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d" gracePeriod=30
Sep 17 19:08:12.822943 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.822476    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:08:12.836793 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836768    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsgz\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.836928 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.837213 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.836990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") pod \"100146e7-01ae-4192-b7c4-e72231811447\" (UID: \"100146e7-01ae-4192-b7c4-e72231811447\") "
Sep 17 19:08:12.837473 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.837382    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:12.837473 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.837405    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:12.839949 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.839917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:12.840066 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.840025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:12.840215 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.840123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.840355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.840329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.840447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.840396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz" (OuterVolumeSpecName: "kube-api-access-2rsgz") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "kube-api-access-2rsgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.847231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.847202    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "100146e7-01ae-4192-b7c4-e72231811447" (UID: "100146e7-01ae-4192-b7c4-e72231811447"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.875579 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.875540    2581 generic.go:356] "Generic (PLEG): container finished" podID="100146e7-01ae-4192-b7c4-e72231811447" containerID="6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d" exitCode=0
Sep 17 19:08:12.875727 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.875690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" event={"ID":"100146e7-01ae-4192-b7c4-e72231811447","Type":"ContainerDied","Data":"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"}
Sep 17 19:08:12.875793 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.875740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd" event={"ID":"100146e7-01ae-4192-b7c4-e72231811447","Type":"ContainerDied","Data":"6a54839a2709c94fa4216cc8a92d858a3f9997041440d298b94c26da82136d1e"}
Sep 17 19:08:12.875793 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.875768    2581 scope.go:117] "RemoveContainer" containerID="6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"
Sep 17 19:08:12.875956 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.875936    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bfb59776-v9qqd"
Sep 17 19:08:12.895571 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.895551    2581 scope.go:117] "RemoveContainer" containerID="6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"
Sep 17 19:08:12.895886 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:08:12.895864    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d\": container with ID starting with 6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d not found: ID does not exist" containerID="6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"
Sep 17 19:08:12.895966 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.895893    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d"} err="failed to get container status \"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d\": rpc error: code = NotFound desc = could not find container \"6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d\": container with ID starting with 6e1f26ab8de4946e30daa785df9e2a5e7ae217b1c4153f3456d13180c986061d not found: ID does not exist"
Sep 17 19:08:12.907558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.907536    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:08:12.912299 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.912277    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bfb59776-v9qqd"]
Sep 17 19:08:12.938867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.938842    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-image-registry-private-configuration\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939017 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939001    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-bound-sa-token\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939114 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939101    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsgz\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-kube-api-access-2rsgz\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939199 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939188    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-registry-certificates\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939280 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939269    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/100146e7-01ae-4192-b7c4-e72231811447-registry-tls\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939361 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939350    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/100146e7-01ae-4192-b7c4-e72231811447-installation-pull-secrets\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939465 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939453    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/100146e7-01ae-4192-b7c4-e72231811447-ca-trust-extracted\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.939544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:12.939533    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/100146e7-01ae-4192-b7c4-e72231811447-trusted-ca\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:08:13.882154 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:13.882124    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975" containerID="91c89784486bdb31b77bb4b6670c6c4c6a0d9f0e65f597a6c5b8b5fa03dd09d0" exitCode=0
Sep 17 19:08:13.882451 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:13.882181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f" event={"ID":"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975","Type":"ContainerDied","Data":"91c89784486bdb31b77bb4b6670c6c4c6a0d9f0e65f597a6c5b8b5fa03dd09d0"}
Sep 17 19:08:13.882545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:13.882527    2581 scope.go:117] "RemoveContainer" containerID="91c89784486bdb31b77bb4b6670c6c4c6a0d9f0e65f597a6c5b8b5fa03dd09d0"
Sep 17 19:08:14.136573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:14.136509    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="100146e7-01ae-4192-b7c4-e72231811447" path="/var/lib/kubelet/pods/100146e7-01ae-4192-b7c4-e72231811447/volumes"
Sep 17 19:08:14.887341 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:08:14.887301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-q272f" event={"ID":"b9dd4cf7-1a53-40eb-8b8d-3146f0fc6975","Type":"ContainerStarted","Data":"11c6e60a404937025dcff4ae8d32663e86597f9fd6cea4508a1527699cefc16b"}
Sep 17 19:11:10.056973 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:11:10.056948    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:11:10.059525 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:11:10.059503    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:11:10.066848 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:11:10.066827    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:13:26.701077 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.701041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"]
Sep 17 19:13:26.701532 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.701345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100146e7-01ae-4192-b7c4-e72231811447" containerName="registry"
Sep 17 19:13:26.701532 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.701356    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="100146e7-01ae-4192-b7c4-e72231811447" containerName="registry"
Sep 17 19:13:26.701532 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.701404    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="100146e7-01ae-4192-b7c4-e72231811447" containerName="registry"
Sep 17 19:13:26.704210 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.704193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.706558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.706535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 19:13:26.707537 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.707515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 19:13:26.707682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.707557    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 19:13:26.707682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.707639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:13:26.707682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.707645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9qxtz\""
Sep 17 19:13:26.707970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.707955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 19:13:26.716271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.716246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"]
Sep 17 19:13:26.837998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.837945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.838186 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.838009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4d2f01c-ef24-4a43-a70a-921caf559235-manager-config\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.838186 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.838100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxxg\" (UniqueName: \"kubernetes.io/projected/b4d2f01c-ef24-4a43-a70a-921caf559235-kube-api-access-vrxxg\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.838186 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.838144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-metrics-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.939399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.939331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.939697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.939450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4d2f01c-ef24-4a43-a70a-921caf559235-manager-config\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.939697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.939498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxxg\" (UniqueName: \"kubernetes.io/projected/b4d2f01c-ef24-4a43-a70a-921caf559235-kube-api-access-vrxxg\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.939697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.939543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-metrics-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.940209 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.940185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b4d2f01c-ef24-4a43-a70a-921caf559235-manager-config\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.942730 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.942703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-metrics-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.942813 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.942735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4d2f01c-ef24-4a43-a70a-921caf559235-cert\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:26.967374 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:26.967295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxxg\" (UniqueName: \"kubernetes.io/projected/b4d2f01c-ef24-4a43-a70a-921caf559235-kube-api-access-vrxxg\") pod \"lws-controller-manager-56c5b7c858-slnkm\" (UID: \"b4d2f01c-ef24-4a43-a70a-921caf559235\") " pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:27.014348 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:27.014318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:27.290074 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:27.289887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"]
Sep 17 19:13:27.292880 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:13:27.292851    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d2f01c_ef24_4a43_a70a_921caf559235.slice/crio-1d50c58a97547d3e40109089a5ec96a73dc52a72c1f5056a17f76671c2171252 WatchSource:0}: Error finding container 1d50c58a97547d3e40109089a5ec96a73dc52a72c1f5056a17f76671c2171252: Status 404 returned error can't find the container with id 1d50c58a97547d3e40109089a5ec96a73dc52a72c1f5056a17f76671c2171252
Sep 17 19:13:27.294768 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:27.294752    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:13:28.046327 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:28.046293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm" event={"ID":"b4d2f01c-ef24-4a43-a70a-921caf559235","Type":"ContainerStarted","Data":"1d50c58a97547d3e40109089a5ec96a73dc52a72c1f5056a17f76671c2171252"}
Sep 17 19:13:30.056744 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:30.056657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm" event={"ID":"b4d2f01c-ef24-4a43-a70a-921caf559235","Type":"ContainerStarted","Data":"9e6e0cdd7c1c1bebbbb39680afcb226eca83f4b63ea9ffe181c3f16b0219765b"}
Sep 17 19:13:30.057124 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:30.056858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:13:30.074916 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:30.074869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm" podStartSLOduration=1.595073378 podStartE2EDuration="4.074852805s" totalImagesPullingTime="2.479779427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:30.072914925 +0000 UTC m=+442.086485086" watchObservedRunningTime="2026-09-17 19:13:30.074852805 +0000 UTC m=+442.088422966"
Sep 17 19:13:41.063110 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:13:41.063082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56c5b7c858-slnkm"
Sep 17 19:14:30.439086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.439046    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:30.442159 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.442144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.444745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.444724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 19:14:30.445068 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.445048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 19:14:30.445152 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.445075    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 19:14:30.445611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.445595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gfjzn\""
Sep 17 19:14:30.452768 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.452750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:30.532428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.532383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:30.584144 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.584117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbtd\" (UniqueName: \"kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.584269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.584156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.684941 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.684914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbtd\" (UniqueName: \"kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.685040 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.684953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.685558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.685542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.697865 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.697815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbtd\" (UniqueName: \"kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd\") pod \"limitador-limitador-8588749977-shpjf\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") " pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:30.752267 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:30.752244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:31.026711 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.026655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:31.030009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:14:31.029984    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f083bd3_3f90_4ecb_8f28_0504ed228def.slice/crio-beb9b3f90732832a85b321b7b17746bf4331d3857c914c0f819e722ba1fcb230 WatchSource:0}: Error finding container beb9b3f90732832a85b321b7b17746bf4331d3857c914c0f819e722ba1fcb230: Status 404 returned error can't find the container with id beb9b3f90732832a85b321b7b17746bf4331d3857c914c0f819e722ba1fcb230
Sep 17 19:14:31.292092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.292023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" event={"ID":"3f083bd3-3f90-4ecb-8f28-0504ed228def","Type":"ContainerStarted","Data":"beb9b3f90732832a85b321b7b17746bf4331d3857c914c0f819e722ba1fcb230"}
Sep 17 19:14:31.378836 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.378808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:31.382875 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.382860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:31.385342 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.385321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t69rq\""
Sep 17 19:14:31.387103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.387079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:31.490730 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.490711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whncg\" (UniqueName: \"kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg\") pod \"authorino-59c974bd48-cg6kc\" (UID: \"b0888a6a-3b06-4dff-b97b-52db0e39b4e8\") " pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:31.591516 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.591466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whncg\" (UniqueName: \"kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg\") pod \"authorino-59c974bd48-cg6kc\" (UID: \"b0888a6a-3b06-4dff-b97b-52db0e39b4e8\") " pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:31.600927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.600903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whncg\" (UniqueName: \"kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg\") pod \"authorino-59c974bd48-cg6kc\" (UID: \"b0888a6a-3b06-4dff-b97b-52db0e39b4e8\") " pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:31.692477 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.692457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:31.964688 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:31.964652    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:31.968379 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:14:31.968349    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0888a6a_3b06_4dff_b97b_52db0e39b4e8.slice/crio-f98de6b15663c78528ad2475fcb48e2ade28d2ecb859f430ad52c25adc73918d WatchSource:0}: Error finding container f98de6b15663c78528ad2475fcb48e2ade28d2ecb859f430ad52c25adc73918d: Status 404 returned error can't find the container with id f98de6b15663c78528ad2475fcb48e2ade28d2ecb859f430ad52c25adc73918d
Sep 17 19:14:32.299443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:32.299348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cg6kc" event={"ID":"b0888a6a-3b06-4dff-b97b-52db0e39b4e8","Type":"ContainerStarted","Data":"f98de6b15663c78528ad2475fcb48e2ade28d2ecb859f430ad52c25adc73918d"}
Sep 17 19:14:34.311440 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:34.311379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" event={"ID":"3f083bd3-3f90-4ecb-8f28-0504ed228def","Type":"ContainerStarted","Data":"f40c39043233cd62955a16284d63cae8fba21aa6616d1e5f838be4a60c441d55"}
Sep 17 19:14:34.311913 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:34.311556    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:35.316836 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:35.316806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cg6kc" event={"ID":"b0888a6a-3b06-4dff-b97b-52db0e39b4e8","Type":"ContainerStarted","Data":"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"}
Sep 17 19:14:35.331884 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:35.331807    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" podStartSLOduration=2.878915867 podStartE2EDuration="5.331790362s" totalImagesPullingTime="2.452874495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:34.327439088 +0000 UTC m=+506.341009250" watchObservedRunningTime="2026-09-17 19:14:35.331790362 +0000 UTC m=+507.345360586"
Sep 17 19:14:35.332314 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:35.332289    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cg6kc" podStartSLOduration=1.8553824049999998 podStartE2EDuration="4.332280857s" totalImagesPullingTime="2.476898452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:35.330617992 +0000 UTC m=+507.344188156" watchObservedRunningTime="2026-09-17 19:14:35.332280857 +0000 UTC m=+507.345851017"
Sep 17 19:14:44.526994 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:44.526964    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:44.527518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:44.527150    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" podUID="3f083bd3-3f90-4ecb-8f28-0504ed228def" containerName="limitador" containerID="cri-o://f40c39043233cd62955a16284d63cae8fba21aa6616d1e5f838be4a60c441d55" gracePeriod=30
Sep 17 19:14:44.527822 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:44.527803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:45.360708 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.360681    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f083bd3-3f90-4ecb-8f28-0504ed228def" containerID="f40c39043233cd62955a16284d63cae8fba21aa6616d1e5f838be4a60c441d55" exitCode=0
Sep 17 19:14:45.360836 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.360758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" event={"ID":"3f083bd3-3f90-4ecb-8f28-0504ed228def","Type":"ContainerDied","Data":"f40c39043233cd62955a16284d63cae8fba21aa6616d1e5f838be4a60c441d55"}
Sep 17 19:14:45.465276 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.465254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:45.601991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.601934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file\") pod \"3f083bd3-3f90-4ecb-8f28-0504ed228def\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") "
Sep 17 19:14:45.601991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.601968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bbtd\" (UniqueName: \"kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd\") pod \"3f083bd3-3f90-4ecb-8f28-0504ed228def\" (UID: \"3f083bd3-3f90-4ecb-8f28-0504ed228def\") "
Sep 17 19:14:45.602322 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.602253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file" (OuterVolumeSpecName: "config-file") pod "3f083bd3-3f90-4ecb-8f28-0504ed228def" (UID: "3f083bd3-3f90-4ecb-8f28-0504ed228def"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:14:45.604252 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.604231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd" (OuterVolumeSpecName: "kube-api-access-5bbtd") pod "3f083bd3-3f90-4ecb-8f28-0504ed228def" (UID: "3f083bd3-3f90-4ecb-8f28-0504ed228def"). InnerVolumeSpecName "kube-api-access-5bbtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:45.702738 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.702702    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbtd\" (UniqueName: \"kubernetes.io/projected/3f083bd3-3f90-4ecb-8f28-0504ed228def-kube-api-access-5bbtd\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:14:45.702738 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:45.702736    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f083bd3-3f90-4ecb-8f28-0504ed228def-config-file\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:14:46.366486 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:46.366455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shpjf" event={"ID":"3f083bd3-3f90-4ecb-8f28-0504ed228def","Type":"ContainerDied","Data":"beb9b3f90732832a85b321b7b17746bf4331d3857c914c0f819e722ba1fcb230"}
Sep 17 19:14:46.366620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:46.366520    2581 scope.go:117] "RemoveContainer" containerID="f40c39043233cd62955a16284d63cae8fba21aa6616d1e5f838be4a60c441d55"
Sep 17 19:14:46.366620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:46.366468    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shpjf"
Sep 17 19:14:46.382370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:46.382174    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:46.383495 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:46.383476    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shpjf"]
Sep 17 19:14:48.136595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:48.136572    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f083bd3-3f90-4ecb-8f28-0504ed228def" path="/var/lib/kubelet/pods/3f083bd3-3f90-4ecb-8f28-0504ed228def/volumes"
Sep 17 19:14:53.328138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.328065    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-t22dg"]
Sep 17 19:14:53.328550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.328361    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f083bd3-3f90-4ecb-8f28-0504ed228def" containerName="limitador"
Sep 17 19:14:53.328550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.328373    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f083bd3-3f90-4ecb-8f28-0504ed228def" containerName="limitador"
Sep 17 19:14:53.328550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.328439    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f083bd3-3f90-4ecb-8f28-0504ed228def" containerName="limitador"
Sep 17 19:14:53.334038 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.334022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.336136 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.336117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 19:14:53.341086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.341064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-t22dg"]
Sep 17 19:14:53.459263 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.459230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6nmq\" (UniqueName: \"kubernetes.io/projected/b1e280b9-d773-47c2-8daf-fdc28f7688df-kube-api-access-r6nmq\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.459399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.459270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1e280b9-d773-47c2-8daf-fdc28f7688df-tls-cert\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.560150 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.560125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6nmq\" (UniqueName: \"kubernetes.io/projected/b1e280b9-d773-47c2-8daf-fdc28f7688df-kube-api-access-r6nmq\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.560245 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.560157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1e280b9-d773-47c2-8daf-fdc28f7688df-tls-cert\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.562684 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.562666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b1e280b9-d773-47c2-8daf-fdc28f7688df-tls-cert\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.568568 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.568545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6nmq\" (UniqueName: \"kubernetes.io/projected/b1e280b9-d773-47c2-8daf-fdc28f7688df-kube-api-access-r6nmq\") pod \"authorino-6d9898bd99-t22dg\" (UID: \"b1e280b9-d773-47c2-8daf-fdc28f7688df\") " pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.643377 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.643334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-t22dg"
Sep 17 19:14:53.910896 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:53.910832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-t22dg"]
Sep 17 19:14:53.913689 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:14:53.913665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e280b9_d773_47c2_8daf_fdc28f7688df.slice/crio-1fcde1883ffaecd4249827f936c03d02a0b5d4a18fbec0f27668c92b090298b9 WatchSource:0}: Error finding container 1fcde1883ffaecd4249827f936c03d02a0b5d4a18fbec0f27668c92b090298b9: Status 404 returned error can't find the container with id 1fcde1883ffaecd4249827f936c03d02a0b5d4a18fbec0f27668c92b090298b9
Sep 17 19:14:54.405368 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:54.405340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-t22dg" event={"ID":"b1e280b9-d773-47c2-8daf-fdc28f7688df","Type":"ContainerStarted","Data":"1fcde1883ffaecd4249827f936c03d02a0b5d4a18fbec0f27668c92b090298b9"}
Sep 17 19:14:55.410690 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.410653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-t22dg" event={"ID":"b1e280b9-d773-47c2-8daf-fdc28f7688df","Type":"ContainerStarted","Data":"dee1148cbfba988a38b3ed9976b29d5956712ecc4faa7a037640d89a24863494"}
Sep 17 19:14:55.427915 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.427865    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-t22dg" podStartSLOduration=1.905110774 podStartE2EDuration="2.427849939s" totalImagesPullingTime="522.739165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:55.427231902 +0000 UTC m=+527.440802064" watchObservedRunningTime="2026-09-17 19:14:55.427849939 +0000 UTC m=+527.441420100"
Sep 17 19:14:55.449607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.449581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:55.449771 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.449750    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cg6kc" podUID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" containerName="authorino" containerID="cri-o://6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5" gracePeriod=30
Sep 17 19:14:55.684528 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.684503    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:55.777731 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.777708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whncg\" (UniqueName: \"kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg\") pod \"b0888a6a-3b06-4dff-b97b-52db0e39b4e8\" (UID: \"b0888a6a-3b06-4dff-b97b-52db0e39b4e8\") "
Sep 17 19:14:55.779783 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.779759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg" (OuterVolumeSpecName: "kube-api-access-whncg") pod "b0888a6a-3b06-4dff-b97b-52db0e39b4e8" (UID: "b0888a6a-3b06-4dff-b97b-52db0e39b4e8"). InnerVolumeSpecName "kube-api-access-whncg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:55.878101 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:55.878079    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whncg\" (UniqueName: \"kubernetes.io/projected/b0888a6a-3b06-4dff-b97b-52db0e39b4e8-kube-api-access-whncg\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:14:56.416902 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.416825    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" containerID="6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5" exitCode=0
Sep 17 19:14:56.416902 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.416867    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cg6kc"
Sep 17 19:14:56.417370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.416906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cg6kc" event={"ID":"b0888a6a-3b06-4dff-b97b-52db0e39b4e8","Type":"ContainerDied","Data":"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"}
Sep 17 19:14:56.417370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.416945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cg6kc" event={"ID":"b0888a6a-3b06-4dff-b97b-52db0e39b4e8","Type":"ContainerDied","Data":"f98de6b15663c78528ad2475fcb48e2ade28d2ecb859f430ad52c25adc73918d"}
Sep 17 19:14:56.417370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.416964    2581 scope.go:117] "RemoveContainer" containerID="6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"
Sep 17 19:14:56.424910 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.424892    2581 scope.go:117] "RemoveContainer" containerID="6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"
Sep 17 19:14:56.425217 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:14:56.425200    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5\": container with ID starting with 6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5 not found: ID does not exist" containerID="6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"
Sep 17 19:14:56.425269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.425231    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5"} err="failed to get container status \"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5\": rpc error: code = NotFound desc = could not find container \"6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5\": container with ID starting with 6c687c053790c5445f633cd08f3ebea7a9545bda25208f9e8317d90b4cae41b5 not found: ID does not exist"
Sep 17 19:14:56.431098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.431077    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:56.434476 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:56.434458    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cg6kc"]
Sep 17 19:14:58.137353 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:14:58.137319    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" path="/var/lib/kubelet/pods/b0888a6a-3b06-4dff-b97b-52db0e39b4e8/volumes"
Sep 17 19:16:10.086729 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:10.086703    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:16:10.087277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:10.086703    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:16:43.320021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.319978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-rgdjp"]
Sep 17 19:16:43.320674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.320324    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" containerName="authorino"
Sep 17 19:16:43.320674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.320337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" containerName="authorino"
Sep 17 19:16:43.320674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.320397    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0888a6a-3b06-4dff-b97b-52db0e39b4e8" containerName="authorino"
Sep 17 19:16:43.323237 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.323218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.325607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.325583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 19:16:43.325730 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.325620    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:16:43.326668 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.326633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 19:16:43.326668 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.326665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qzlfl\""
Sep 17 19:16:43.327030 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.327000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:16:43.331474 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.331448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-rgdjp"]
Sep 17 19:16:43.460893 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.460848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk97t\" (UniqueName: \"kubernetes.io/projected/142d0912-1c13-4a0d-a83a-959fa5ea2012-kube-api-access-lk97t\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.460893 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.460891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.461163 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.460916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.561504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.561459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk97t\" (UniqueName: \"kubernetes.io/projected/142d0912-1c13-4a0d-a83a-959fa5ea2012-kube-api-access-lk97t\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.561716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.561514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.561716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.561551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:43.561716 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:16:43.561658    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 19:16:43.561716 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:16:43.561715    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 19:16:43.561925 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:16:43.561722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert podName:142d0912-1c13-4a0d-a83a-959fa5ea2012 nodeName:}" failed. No retries permitted until 2026-09-17 19:16:44.061705826 +0000 UTC m=+636.075275965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert") pod "odh-model-controller-7b966d7984-rgdjp" (UID: "142d0912-1c13-4a0d-a83a-959fa5ea2012") : secret "odh-model-controller-webhook-cert" not found
Sep 17 19:16:43.561925 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:16:43.561763    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs podName:142d0912-1c13-4a0d-a83a-959fa5ea2012 nodeName:}" failed. No retries permitted until 2026-09-17 19:16:44.061746595 +0000 UTC m=+636.075316738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs") pod "odh-model-controller-7b966d7984-rgdjp" (UID: "142d0912-1c13-4a0d-a83a-959fa5ea2012") : secret "odh-model-controller-metrics-tls" not found
Sep 17 19:16:43.572202 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:43.572134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk97t\" (UniqueName: \"kubernetes.io/projected/142d0912-1c13-4a0d-a83a-959fa5ea2012-kube-api-access-lk97t\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.067004 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.066962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.067004 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.067008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.069898 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.069864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-cert\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.069898 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.069888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142d0912-1c13-4a0d-a83a-959fa5ea2012-metrics-certs\") pod \"odh-model-controller-7b966d7984-rgdjp\" (UID: \"142d0912-1c13-4a0d-a83a-959fa5ea2012\") " pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.237889 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.237845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:44.517636 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.517588    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-rgdjp"]
Sep 17 19:16:44.519905 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:16:44.519877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142d0912_1c13_4a0d_a83a_959fa5ea2012.slice/crio-eb055b876df0ee83a98bbd76d5657bf06237009c7a73227ad108e0d5a0a78a13 WatchSource:0}: Error finding container eb055b876df0ee83a98bbd76d5657bf06237009c7a73227ad108e0d5a0a78a13: Status 404 returned error can't find the container with id eb055b876df0ee83a98bbd76d5657bf06237009c7a73227ad108e0d5a0a78a13
Sep 17 19:16:44.849269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:44.849166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-rgdjp" event={"ID":"142d0912-1c13-4a0d-a83a-959fa5ea2012","Type":"ContainerStarted","Data":"eb055b876df0ee83a98bbd76d5657bf06237009c7a73227ad108e0d5a0a78a13"}
Sep 17 19:16:47.866340 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:47.866300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-rgdjp" event={"ID":"142d0912-1c13-4a0d-a83a-959fa5ea2012","Type":"ContainerStarted","Data":"e8916ccb51748eb4121f664ea278d3d503a9666445eb94699d1ac734143deca9"}
Sep 17 19:16:47.868727 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:47.866511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:16:47.886223 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:47.886124    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-rgdjp" podStartSLOduration=2.012341516 podStartE2EDuration="4.886102282s" totalImagesPullingTime="2.873760766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:47.884526535 +0000 UTC m=+639.898096698" watchObservedRunningTime="2026-09-17 19:16:47.886102282 +0000 UTC m=+639.899672497"
Sep 17 19:16:58.874148 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:16:58.874115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-rgdjp"
Sep 17 19:18:43.369437 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.369378    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:18:43.373207 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.373182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.377388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.377365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-2dmbc\""
Sep 17 19:18:43.377530 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.377370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 19:18:43.377784 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.377761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-wj8g4\""
Sep 17 19:18:43.377897 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.377806    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 19:18:43.377897 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.377850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 19:18:43.389222 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.389194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:18:43.430105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.430070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.430292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.430135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqqh\" (UniqueName: \"kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.531235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.531197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqqh\" (UniqueName: \"kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.531426 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.531260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.533960 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.533940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.548551 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.548519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqqh\" (UniqueName: \"kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.686028 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.685995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:43.973022 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.972929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:18:43.980609 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:18:43.980572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1e3866_ac8f_43e7_8025_8e3e66ed78c2.slice/crio-2992364bd306dab318755aa32c4ab1eda85040d49e76c1cce75403429d359349 WatchSource:0}: Error finding container 2992364bd306dab318755aa32c4ab1eda85040d49e76c1cce75403429d359349: Status 404 returned error can't find the container with id 2992364bd306dab318755aa32c4ab1eda85040d49e76c1cce75403429d359349
Sep 17 19:18:43.983178 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:43.983157    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:18:44.373845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:44.373746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" event={"ID":"af1e3866-ac8f-43e7-8025-8e3e66ed78c2","Type":"ContainerStarted","Data":"2992364bd306dab318755aa32c4ab1eda85040d49e76c1cce75403429d359349"}
Sep 17 19:18:46.384998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:46.384963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" event={"ID":"af1e3866-ac8f-43e7-8025-8e3e66ed78c2","Type":"ContainerStarted","Data":"2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff"}
Sep 17 19:18:46.385394 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:46.385028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:18:46.400090 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:18:46.400018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" podStartSLOduration=1.15732208 podStartE2EDuration="3.399986066s" totalImagesPullingTime="2.242663986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:46.399956604 +0000 UTC m=+758.413526766" watchObservedRunningTime="2026-09-17 19:18:46.399986066 +0000 UTC m=+758.413556229"
Sep 17 19:19:16.573484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.573445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:16.581541 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.581515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.584569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.584545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 19:19:16.585219 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.585158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 19:19:16.585360 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.585230    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 19:19:16.585852 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.585827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-75pbr\""
Sep 17 19:19:16.586437 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.586396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:16.601818 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.601792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.601937 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.601823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.601937 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.601844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhsl\" (UniqueName: \"kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.601937 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.601931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.602067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.601972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.602067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.602003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702441 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702441 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhsl\" (UniqueName: \"kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.702935 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.703021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.703021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.702962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.705406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.705380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.705677 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.705660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.711119 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.711098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhsl\" (UniqueName: \"kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl\") pod \"fc-smoke-test-kserve-7f4687897-rzg27\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.793799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.793761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:19:16.797802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.797784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.800041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.800020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-nbrxk\""
Sep 17 19:19:16.802991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.802971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nbt\" (UniqueName: \"kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.803097 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.803036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.807846 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.807823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:19:16.893394 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.893307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:16.904022 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.903988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.904147 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.904058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nbt\" (UniqueName: \"kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.906589 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.906567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:16.916920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:16.916892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nbt\" (UniqueName: \"kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:17.108405 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.108369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:17.168367 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.168062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:17.172009 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:19:17.171972    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08de87f5_2e07_4f62_8103_d087168c9499.slice/crio-aa0f8eccbb9e907c47c7f7a69be66cdfdae74154b063f8f4187254a3fafc10ac WatchSource:0}: Error finding container aa0f8eccbb9e907c47c7f7a69be66cdfdae74154b063f8f4187254a3fafc10ac: Status 404 returned error can't find the container with id aa0f8eccbb9e907c47c7f7a69be66cdfdae74154b063f8f4187254a3fafc10ac
Sep 17 19:19:17.383515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.383466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:19:17.386462 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:19:17.386411    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f150790_1876_43aa_a1b9_0aad8dc44f88.slice/crio-1088489f5981cfb902dc0bf5498e179da4202e65d03e54cae5ab26852343bde7 WatchSource:0}: Error finding container 1088489f5981cfb902dc0bf5498e179da4202e65d03e54cae5ab26852343bde7: Status 404 returned error can't find the container with id 1088489f5981cfb902dc0bf5498e179da4202e65d03e54cae5ab26852343bde7
Sep 17 19:19:17.391918 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.391893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:19:17.525773 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.525709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" event={"ID":"0f150790-1876-43aa-a1b9-0aad8dc44f88","Type":"ContainerStarted","Data":"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"}
Sep 17 19:19:17.525962 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.525828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" event={"ID":"0f150790-1876-43aa-a1b9-0aad8dc44f88","Type":"ContainerStarted","Data":"1088489f5981cfb902dc0bf5498e179da4202e65d03e54cae5ab26852343bde7"}
Sep 17 19:19:17.525962 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.525873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:17.527452 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.527393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" event={"ID":"08de87f5-2e07-4f62-8103-d087168c9499","Type":"ContainerStarted","Data":"aa0f8eccbb9e907c47c7f7a69be66cdfdae74154b063f8f4187254a3fafc10ac"}
Sep 17 19:19:17.550146 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:17.549950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podStartSLOduration=1.549927239 podStartE2EDuration="1.549927239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:17.547880803 +0000 UTC m=+789.561450964" watchObservedRunningTime="2026-09-17 19:19:17.549927239 +0000 UTC m=+789.563497401"
Sep 17 19:19:19.540156 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:19.540114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" event={"ID":"08de87f5-2e07-4f62-8103-d087168c9499","Type":"ContainerStarted","Data":"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"}
Sep 17 19:19:19.561177 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:19.561116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" podStartSLOduration=1.7728385599999998 podStartE2EDuration="3.561100162s" totalImagesPullingTime="1.788261602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:19.558633381 +0000 UTC m=+791.572203542" watchObservedRunningTime="2026-09-17 19:19:19.561100162 +0000 UTC m=+791.574670348"
Sep 17 19:19:19.894031 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:19.893896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:26.894376 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:26.894337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:26.906949 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:26.906922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:27.595542 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:27.595511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:41.639287 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:41.639250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:19:47.392237 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:47.392198    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:48.535194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:48.535162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:19:52.602296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.602260    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 19:19:52.602658 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:19:52.602299    2581 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main" containerID="cri-o://2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff"
Sep 17 19:19:52.602658 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.602368    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main" containerID="cri-o://2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff" gracePeriod=50
Sep 17 19:19:52.708371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.708339    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm_af1e3866-ac8f-43e7-8025-8e3e66ed78c2/main/0.log"
Sep 17 19:19:52.708553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.708381    2581 generic.go:356] "Generic (PLEG): container finished" podID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerID="2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff" exitCode=1
Sep 17 19:19:52.708553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.708439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" event={"ID":"af1e3866-ac8f-43e7-8025-8e3e66ed78c2","Type":"ContainerDied","Data":"2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff"}
Sep 17 19:19:52.743837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.743816    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm_af1e3866-ac8f-43e7-8025-8e3e66ed78c2/main/0.log"
Sep 17 19:19:52.746238 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.746224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:19:52.920977 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.920949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs\") pod \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") "
Sep 17 19:19:52.921150 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.920984    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lqqh\" (UniqueName: \"kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh\") pod \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\" (UID: \"af1e3866-ac8f-43e7-8025-8e3e66ed78c2\") "
Sep 17 19:19:52.923381 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.923348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh" (OuterVolumeSpecName: "kube-api-access-8lqqh") pod "af1e3866-ac8f-43e7-8025-8e3e66ed78c2" (UID: "af1e3866-ac8f-43e7-8025-8e3e66ed78c2"). InnerVolumeSpecName "kube-api-access-8lqqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:52.923512 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:52.923378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af1e3866-ac8f-43e7-8025-8e3e66ed78c2" (UID: "af1e3866-ac8f-43e7-8025-8e3e66ed78c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:53.021882 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.021848    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:53.021882 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.021876    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqqh\" (UniqueName: \"kubernetes.io/projected/af1e3866-ac8f-43e7-8025-8e3e66ed78c2-kube-api-access-8lqqh\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:53.714842 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.714815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm_af1e3866-ac8f-43e7-8025-8e3e66ed78c2/main/0.log"
Sep 17 19:19:53.715282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.714914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"
Sep 17 19:19:53.715282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.714941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm" event={"ID":"af1e3866-ac8f-43e7-8025-8e3e66ed78c2","Type":"ContainerDied","Data":"2992364bd306dab318755aa32c4ab1eda85040d49e76c1cce75403429d359349"}
Sep 17 19:19:53.715282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.714994    2581 scope.go:117] "RemoveContainer" containerID="2b5aad072b79c767f09d3b7fbd7dbba4133103adf45aaf371bfa8b1d81ae0fff"
Sep 17 19:19:53.736507 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.736479    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:19:53.742212 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:53.742175    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfggm"]
Sep 17 19:19:54.137593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:54.137516    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" path="/var/lib/kubelet/pods/af1e3866-ac8f-43e7-8025-8e3e66ed78c2/volumes"
Sep 17 19:19:55.029336 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.029280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:55.031680 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.031634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:19:55.044306    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696" cmd=["/bin/sleep","15"]
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.044377    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" message="" err=<
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:19:55.044396    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" podUID="08de87f5-2e07-4f62-8103-d087168c9499" containerName="main" containerID="cri-o://9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"
Sep 17 19:19:55.044519 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.044476    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" podUID="08de87f5-2e07-4f62-8103-d087168c9499" containerName="main" containerID="cri-o://9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696" gracePeriod=60
Sep 17 19:19:55.406723 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.406696    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:55.545997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.545959    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfhsl\" (UniqueName: \"kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546137 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546137 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546135    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm\") pod \"08de87f5-2e07-4f62-8103-d087168c9499\" (UID: \"08de87f5-2e07-4f62-8103-d087168c9499\") "
Sep 17 19:19:55.546405 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546251    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:55.546405 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache" (OuterVolumeSpecName: "model-cache") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:55.546535 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home" (OuterVolumeSpecName: "home") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:55.546579 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546546    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.546579 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546563    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.546647 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.546589    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.548570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.548538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm" (OuterVolumeSpecName: "dshm") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:55.548570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.548552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl" (OuterVolumeSpecName: "kube-api-access-dfhsl") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "kube-api-access-dfhsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:55.548722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.548693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08de87f5-2e07-4f62-8103-d087168c9499" (UID: "08de87f5-2e07-4f62-8103-d087168c9499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:55.647837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.647765    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08de87f5-2e07-4f62-8103-d087168c9499-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.647837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.647795    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhsl\" (UniqueName: \"kubernetes.io/projected/08de87f5-2e07-4f62-8103-d087168c9499-kube-api-access-dfhsl\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.647837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.647806    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08de87f5-2e07-4f62-8103-d087168c9499-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:19:55.726310 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.726270    2581 generic.go:356] "Generic (PLEG): container finished" podID="08de87f5-2e07-4f62-8103-d087168c9499" containerID="9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696" exitCode=0
Sep 17 19:19:55.726478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.726317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"
Sep 17 19:19:55.726478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.726350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" event={"ID":"08de87f5-2e07-4f62-8103-d087168c9499","Type":"ContainerDied","Data":"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"}
Sep 17 19:19:55.726478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.726409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27" event={"ID":"08de87f5-2e07-4f62-8103-d087168c9499","Type":"ContainerDied","Data":"aa0f8eccbb9e907c47c7f7a69be66cdfdae74154b063f8f4187254a3fafc10ac"}
Sep 17 19:19:55.726478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.726455    2581 scope.go:117] "RemoveContainer" containerID="9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"
Sep 17 19:19:55.735369 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.735349    2581 scope.go:117] "RemoveContainer" containerID="9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"
Sep 17 19:19:55.735676 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:19:55.735659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696\": container with ID starting with 9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696 not found: ID does not exist" containerID="9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"
Sep 17 19:19:55.735737 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.735686    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696"} err="failed to get container status \"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696\": rpc error: code = NotFound desc = could not find container \"9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696\": container with ID starting with 9380458c6856d26d471106bf1c6672f2c8333acd2c9efa01f6400a0683e36696 not found: ID does not exist"
Sep 17 19:19:55.750834 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.750781    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:55.752524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:55.752499    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-rzg27"]
Sep 17 19:19:56.137261 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:56.137228    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08de87f5-2e07-4f62-8103-d087168c9499" path="/var/lib/kubelet/pods/08de87f5-2e07-4f62-8103-d087168c9499/volumes"
Sep 17 19:19:58.535499 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:19:58.535456    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:08.535407 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:08.535365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:10.055025 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:10.054987    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" containerID="cri-o://9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892" gracePeriod=45
Sep 17 19:20:18.534891 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:18.534846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:18.535384 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:18.534974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:20:28.535463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:28.535397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:35.173904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.173863    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:20:35.174893 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.174867    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08de87f5-2e07-4f62-8103-d087168c9499" containerName="main"
Sep 17 19:20:35.174893 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.174895    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="08de87f5-2e07-4f62-8103-d087168c9499" containerName="main"
Sep 17 19:20:35.175039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.174911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main"
Sep 17 19:20:35.175039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.174920    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main"
Sep 17 19:20:35.175039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.175009    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1e3866-ac8f-43e7-8025-8e3e66ed78c2" containerName="main"
Sep 17 19:20:35.175039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.175019    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="08de87f5-2e07-4f62-8103-d087168c9499" containerName="main"
Sep 17 19:20:35.178189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.178168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.182141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.181643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-tpnbg\""
Sep 17 19:20:35.182141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.181677    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 19:20:35.182141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.181703    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 19:20:35.182141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.181704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 19:20:35.182141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.181863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-sx2z8\""
Sep 17 19:20:35.186997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.186974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:20:35.256208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.256165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.256409 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.256356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.256409 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.256388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.256568 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.256490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.357866 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.357826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.358047 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.357879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.358047 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.357906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.358047 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.357939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.358257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.358238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.358307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.358286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.360679 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.360660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.366567 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.366547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.490593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.490500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.778202 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:20:35.778161    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c551460_b29b_4584_bc35_2ad6654046ad.slice/crio-acaf443af8b964b7b4add6acb8a3489c433bd41b516b2a687e5071940dada6a5 WatchSource:0}: Error finding container acaf443af8b964b7b4add6acb8a3489c433bd41b516b2a687e5071940dada6a5: Status 404 returned error can't find the container with id acaf443af8b964b7b4add6acb8a3489c433bd41b516b2a687e5071940dada6a5
Sep 17 19:20:35.791927 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.791850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:20:35.907698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.907654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" event={"ID":"3c551460-b29b-4584-bc35-2ad6654046ad","Type":"ContainerStarted","Data":"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"}
Sep 17 19:20:35.907899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.907726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" event={"ID":"3c551460-b29b-4584-bc35-2ad6654046ad","Type":"ContainerStarted","Data":"acaf443af8b964b7b4add6acb8a3489c433bd41b516b2a687e5071940dada6a5"}
Sep 17 19:20:35.908586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.908564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:20:35.926028 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:35.925962    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podStartSLOduration=0.925938909 podStartE2EDuration="925.938909ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:35.925088268 +0000 UTC m=+867.938658457" watchObservedRunningTime="2026-09-17 19:20:35.925938909 +0000 UTC m=+867.939509072"
Sep 17 19:20:38.535081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:38.535036    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:40.304199 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.304172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:20:40.405547 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.405444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs\") pod \"0f150790-1876-43aa-a1b9-0aad8dc44f88\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") "
Sep 17 19:20:40.405547 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.405500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2nbt\" (UniqueName: \"kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt\") pod \"0f150790-1876-43aa-a1b9-0aad8dc44f88\" (UID: \"0f150790-1876-43aa-a1b9-0aad8dc44f88\") "
Sep 17 19:20:40.407797 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.407771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt" (OuterVolumeSpecName: "kube-api-access-n2nbt") pod "0f150790-1876-43aa-a1b9-0aad8dc44f88" (UID: "0f150790-1876-43aa-a1b9-0aad8dc44f88"). InnerVolumeSpecName "kube-api-access-n2nbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:40.407797 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.407771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f150790-1876-43aa-a1b9-0aad8dc44f88" (UID: "0f150790-1876-43aa-a1b9-0aad8dc44f88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:40.506570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.506538    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nbt\" (UniqueName: \"kubernetes.io/projected/0f150790-1876-43aa-a1b9-0aad8dc44f88-kube-api-access-n2nbt\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:20:40.506570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.506566    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f150790-1876-43aa-a1b9-0aad8dc44f88-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:20:40.932039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.931995    2581 generic.go:356] "Generic (PLEG): container finished" podID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerID="9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892" exitCode=0
Sep 17 19:20:40.932207 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.932051    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"
Sep 17 19:20:40.932207 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.932080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" event={"ID":"0f150790-1876-43aa-a1b9-0aad8dc44f88","Type":"ContainerDied","Data":"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"}
Sep 17 19:20:40.932207 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.932139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz" event={"ID":"0f150790-1876-43aa-a1b9-0aad8dc44f88","Type":"ContainerDied","Data":"1088489f5981cfb902dc0bf5498e179da4202e65d03e54cae5ab26852343bde7"}
Sep 17 19:20:40.932207 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.932168    2581 scope.go:117] "RemoveContainer" containerID="9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"
Sep 17 19:20:40.943143 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.942981    2581 scope.go:117] "RemoveContainer" containerID="9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"
Sep 17 19:20:40.943669 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:20:40.943636    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892\": container with ID starting with 9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892 not found: ID does not exist" containerID="9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"
Sep 17 19:20:40.943760 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.943677    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892"} err="failed to get container status \"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892\": rpc error: code = NotFound desc = could not find container \"9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892\": container with ID starting with 9958384e82fad8cd5afbcdff8af1890a283fa807cae62a8cdf7132e1f9394892 not found: ID does not exist"
Sep 17 19:20:40.958769 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.958738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:20:40.966133 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:40.966105    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-7fjqz"]
Sep 17 19:20:42.137223 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:20:42.137191    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" path="/var/lib/kubelet/pods/0f150790-1876-43aa-a1b9-0aad8dc44f88/volumes"
Sep 17 19:21:02.724843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.724744    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:21:02.725307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.725081    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main"
Sep 17 19:21:02.725307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.725092    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main"
Sep 17 19:21:02.725307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.725150    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f150790-1876-43aa-a1b9-0aad8dc44f88" containerName="main"
Sep 17 19:21:02.727728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.727705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:02.731181 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.731155    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 19:21:02.731323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.731253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-m2gbz\""
Sep 17 19:21:02.731398 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.731200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 19:21:02.731398 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.731219    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-22qhc\""
Sep 17 19:21:02.731550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.731196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 19:21:02.739191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.739169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:21:02.900467 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.900398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:02.900636 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:02.900583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gbc\" (UniqueName: \"kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.002088 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.001911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.002088 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.002039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gbc\" (UniqueName: \"kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.005137 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.005098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.011629 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.011599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gbc\" (UniqueName: \"kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.038632 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.038591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:03.312607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:03.312582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:21:03.314747 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:21:03.314717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47284ffb_ae9e_462a_9a91_bfea7bcdb38c.slice/crio-65a9bf13af7d42df4ceee4d1dd4cc9eda3bd09c0422ea42daec3a0564d321184 WatchSource:0}: Error finding container 65a9bf13af7d42df4ceee4d1dd4cc9eda3bd09c0422ea42daec3a0564d321184: Status 404 returned error can't find the container with id 65a9bf13af7d42df4ceee4d1dd4cc9eda3bd09c0422ea42daec3a0564d321184
Sep 17 19:21:04.037502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:04.037464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" event={"ID":"47284ffb-ae9e-462a-9a91-bfea7bcdb38c","Type":"ContainerStarted","Data":"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"}
Sep 17 19:21:04.037948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:04.037527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" event={"ID":"47284ffb-ae9e-462a-9a91-bfea7bcdb38c","Type":"ContainerStarted","Data":"65a9bf13af7d42df4ceee4d1dd4cc9eda3bd09c0422ea42daec3a0564d321184"}
Sep 17 19:21:04.037948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:04.037660    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:04.056581 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:04.056531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podStartSLOduration=2.05651423 podStartE2EDuration="2.05651423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:04.055507231 +0000 UTC m=+896.069077394" watchObservedRunningTime="2026-09-17 19:21:04.05651423 +0000 UTC m=+896.070084390"
Sep 17 19:21:07.918143 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:07.918112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:21:10.114802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:10.114765    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:21:10.116196 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:10.116171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:21:13.167998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:13.167962    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:21:17.917352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:17.917309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:27.917724 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:27.917687    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:28.189323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:28.189257    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" containerID="cri-o://fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634" gracePeriod=45
Sep 17 19:21:35.043601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:35.043571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:21:37.917797 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:37.917761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:37.918253 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:37.917876    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:21:41.285079 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:41.285042    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:21:45.044489 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:45.044447    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:47.917672 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:47.917510    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:55.044378 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:55.044339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:56.306441 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:56.306379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" containerID="cri-o://5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e" gracePeriod=45
Sep 17 19:21:57.917731 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:57.917694    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:21:58.436832 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.436803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:21:58.572040 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.571933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache\") pod \"3c551460-b29b-4584-bc35-2ad6654046ad\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") "
Sep 17 19:21:58.572040 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.571998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq\") pod \"3c551460-b29b-4584-bc35-2ad6654046ad\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") "
Sep 17 19:21:58.572480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.572043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs\") pod \"3c551460-b29b-4584-bc35-2ad6654046ad\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") "
Sep 17 19:21:58.572564 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.572471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3c551460-b29b-4584-bc35-2ad6654046ad" (UID: "3c551460-b29b-4584-bc35-2ad6654046ad"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:58.572564 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.572530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache\") pod \"3c551460-b29b-4584-bc35-2ad6654046ad\" (UID: \"3c551460-b29b-4584-bc35-2ad6654046ad\") "
Sep 17 19:21:58.573280 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.573072    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "3c551460-b29b-4584-bc35-2ad6654046ad" (UID: "3c551460-b29b-4584-bc35-2ad6654046ad"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:58.573464 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.573442    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-cachi2-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:21:58.573569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.573471    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c551460-b29b-4584-bc35-2ad6654046ad-tokenizer-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:21:58.579316 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.579267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq" (OuterVolumeSpecName: "kube-api-access-5qbdq") pod "3c551460-b29b-4584-bc35-2ad6654046ad" (UID: "3c551460-b29b-4584-bc35-2ad6654046ad"). InnerVolumeSpecName "kube-api-access-5qbdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:58.580958 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.580928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c551460-b29b-4584-bc35-2ad6654046ad" (UID: "3c551460-b29b-4584-bc35-2ad6654046ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:58.674799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.674760    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbdq\" (UniqueName: \"kubernetes.io/projected/3c551460-b29b-4584-bc35-2ad6654046ad-kube-api-access-5qbdq\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:21:58.674799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:58.674799    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c551460-b29b-4584-bc35-2ad6654046ad-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:21:59.281797 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.281759    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c551460-b29b-4584-bc35-2ad6654046ad" containerID="fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634" exitCode=0
Sep 17 19:21:59.282281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.281822    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"
Sep 17 19:21:59.282281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.281848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" event={"ID":"3c551460-b29b-4584-bc35-2ad6654046ad","Type":"ContainerDied","Data":"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"}
Sep 17 19:21:59.282281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.281907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f" event={"ID":"3c551460-b29b-4584-bc35-2ad6654046ad","Type":"ContainerDied","Data":"acaf443af8b964b7b4add6acb8a3489c433bd41b516b2a687e5071940dada6a5"}
Sep 17 19:21:59.282281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.281933    2581 scope.go:117] "RemoveContainer" containerID="fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"
Sep 17 19:21:59.290948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.290928    2581 scope.go:117] "RemoveContainer" containerID="fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"
Sep 17 19:21:59.291272 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:21:59.291252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634\": container with ID starting with fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634 not found: ID does not exist" containerID="fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"
Sep 17 19:21:59.291348 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.291283    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634"} err="failed to get container status \"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634\": rpc error: code = NotFound desc = could not find container \"fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634\": container with ID starting with fb1de2259100dcd50838c9f08cfad60c2a103c4ba3d22d2a1880c4a970441634 not found: ID does not exist"
Sep 17 19:21:59.303246 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.303210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:21:59.307055 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:21:59.307026    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcf7k5f"]
Sep 17 19:22:00.137127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:00.137098    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" path="/var/lib/kubelet/pods/3c551460-b29b-4584-bc35-2ad6654046ad/volumes"
Sep 17 19:22:05.043701 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:05.043655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:05.044102 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:05.043770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:22:09.440150 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.440117    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:22:09.440560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.440467    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main"
Sep 17 19:22:09.440560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.440479    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main"
Sep 17 19:22:09.440560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.440557    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c551460-b29b-4584-bc35-2ad6654046ad" containerName="main"
Sep 17 19:22:09.445118 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.445094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.447879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.447855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 19:22:09.448008 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.447855    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 19:22:09.448008 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.447856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cch6v\""
Sep 17 19:22:09.448008 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.447856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 19:22:09.456871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.456846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:22:09.468553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.468745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.468745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.468745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwb98\" (UniqueName: \"kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.468745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.468745 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.468742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.569863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.569817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.569892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.569923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.569975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwb98\" (UniqueName: \"kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.570092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570233 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.570139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570359 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.570335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570458 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.570386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.570514 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.570478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.572349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.572323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.572633 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.572616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.578879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.578858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwb98\" (UniqueName: \"kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.647036 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.646992    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:22:09.650551 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.650521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.653188 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.653164    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-d24l9\""
Sep 17 19:22:09.659948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.659917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:22:09.671441 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.671393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjnh\" (UniqueName: \"kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.671599 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.671488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.755497 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.755370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:09.772870 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.772838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.773009 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.772913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjnh\" (UniqueName: \"kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.775880 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.775848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.783374 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.783334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjnh\" (UniqueName: \"kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:09.965676 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:09.965631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:10.033819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.033794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:22:10.036102 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:22:10.036077    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb619896_252e_4ebf_8c0b_dfabf6782eed.slice/crio-fa3d6d4a73d425b210901eb4bdf54e4c413319bc117cdebd27bdffc01ad27231 WatchSource:0}: Error finding container fa3d6d4a73d425b210901eb4bdf54e4c413319bc117cdebd27bdffc01ad27231: Status 404 returned error can't find the container with id fa3d6d4a73d425b210901eb4bdf54e4c413319bc117cdebd27bdffc01ad27231
Sep 17 19:22:10.237481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.237441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:22:10.241286 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:22:10.241258    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb63de1_f5db_47f2_8928_8a9aaa454b30.slice/crio-4528007c5ef4f9ef2d593f5835af0b37444e402d4e155073d41fa969f2498c0e WatchSource:0}: Error finding container 4528007c5ef4f9ef2d593f5835af0b37444e402d4e155073d41fa969f2498c0e: Status 404 returned error can't find the container with id 4528007c5ef4f9ef2d593f5835af0b37444e402d4e155073d41fa969f2498c0e
Sep 17 19:22:10.334161 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.334115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" event={"ID":"efb63de1-f5db-47f2-8928-8a9aaa454b30","Type":"ContainerStarted","Data":"d08e37291c011a5901cf20397b6f1d453a1be1b77d7b08e3a0329ebdca5242d1"}
Sep 17 19:22:10.334366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.334184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" event={"ID":"efb63de1-f5db-47f2-8928-8a9aaa454b30","Type":"ContainerStarted","Data":"4528007c5ef4f9ef2d593f5835af0b37444e402d4e155073d41fa969f2498c0e"}
Sep 17 19:22:10.334366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.334294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:22:10.336235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.336200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" event={"ID":"bb619896-252e-4ebf-8c0b-dfabf6782eed","Type":"ContainerStarted","Data":"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"}
Sep 17 19:22:10.336380 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.336271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" event={"ID":"bb619896-252e-4ebf-8c0b-dfabf6782eed","Type":"ContainerStarted","Data":"fa3d6d4a73d425b210901eb4bdf54e4c413319bc117cdebd27bdffc01ad27231"}
Sep 17 19:22:10.356320 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.356260    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podStartSLOduration=1.356238866 podStartE2EDuration="1.356238866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:10.354982922 +0000 UTC m=+962.368553096" watchObservedRunningTime="2026-09-17 19:22:10.356238866 +0000 UTC m=+962.369809027"
Sep 17 19:22:10.376517 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.376448    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" podStartSLOduration=1.376432495 podStartE2EDuration="1.376432495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:10.37438554 +0000 UTC m=+962.387955701" watchObservedRunningTime="2026-09-17 19:22:10.376432495 +0000 UTC m=+962.390002653"
Sep 17 19:22:10.756527 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:10.756482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:15.043619 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:15.043581    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:19.755595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:19.755558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:19.768243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:19.768209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:20.391128 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:20.391096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:22:25.044562 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:25.044520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:26.572342 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.572278    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:22:26.610487 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.610451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs\") pod \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") "
Sep 17 19:22:26.610657 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.610532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7gbc\" (UniqueName: \"kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc\") pod \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\" (UID: \"47284ffb-ae9e-462a-9a91-bfea7bcdb38c\") "
Sep 17 19:22:26.613092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.613046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc" (OuterVolumeSpecName: "kube-api-access-z7gbc") pod "47284ffb-ae9e-462a-9a91-bfea7bcdb38c" (UID: "47284ffb-ae9e-462a-9a91-bfea7bcdb38c"). InnerVolumeSpecName "kube-api-access-z7gbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:26.613231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.613116    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47284ffb-ae9e-462a-9a91-bfea7bcdb38c" (UID: "47284ffb-ae9e-462a-9a91-bfea7bcdb38c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:26.712120 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.712085    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:22:26.712120 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:26.712114    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gbc\" (UniqueName: \"kubernetes.io/projected/47284ffb-ae9e-462a-9a91-bfea7bcdb38c-kube-api-access-z7gbc\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:22:27.414829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.414784    2581 generic.go:356] "Generic (PLEG): container finished" podID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerID="5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e" exitCode=0
Sep 17 19:22:27.415058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.414836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"
Sep 17 19:22:27.415058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.414840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" event={"ID":"47284ffb-ae9e-462a-9a91-bfea7bcdb38c","Type":"ContainerDied","Data":"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"}
Sep 17 19:22:27.415058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.415006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2" event={"ID":"47284ffb-ae9e-462a-9a91-bfea7bcdb38c","Type":"ContainerDied","Data":"65a9bf13af7d42df4ceee4d1dd4cc9eda3bd09c0422ea42daec3a0564d321184"}
Sep 17 19:22:27.415058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.415037    2581 scope.go:117] "RemoveContainer" containerID="5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"
Sep 17 19:22:27.424986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.424853    2581 scope.go:117] "RemoveContainer" containerID="5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"
Sep 17 19:22:27.425173 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:22:27.425154    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e\": container with ID starting with 5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e not found: ID does not exist" containerID="5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"
Sep 17 19:22:27.425215 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.425180    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e"} err="failed to get container status \"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e\": rpc error: code = NotFound desc = could not find container \"5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e\": container with ID starting with 5bac7e69393250a66f9e09a3876c5bff1782868efe1ae5835be664d9e8b3721e not found: ID does not exist"
Sep 17 19:22:27.438682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.438637    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:22:27.442148 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:27.442113    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-69qv2"]
Sep 17 19:22:28.137234 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:28.137198    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" path="/var/lib/kubelet/pods/47284ffb-ae9e-462a-9a91-bfea7bcdb38c/volumes"
Sep 17 19:22:41.342722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:22:41.342690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:23:08.589403 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.589365    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:23:08.593886 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.593856    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:23:08.601128 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:23:08.601076    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:23:08.601128 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.601128 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.601128 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25" cmd=["/bin/sleep","15"]
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.601143    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" message="" err=<
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:23:08.601161    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerName="main" containerID="cri-o://90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"
Sep 17 19:23:08.601500 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.601209    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerName="main" containerID="cri-o://90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25" gracePeriod=60
Sep 17 19:23:08.953767 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.953740    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:23:08.985022 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.984992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985065    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwb98\" (UniqueName: \"kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs\") pod \"bb619896-252e-4ebf-8c0b-dfabf6782eed\" (UID: \"bb619896-252e-4ebf-8c0b-dfabf6782eed\") "
Sep 17 19:23:08.985480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:08.985480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:08.985480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home" (OuterVolumeSpecName: "home") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:08.985892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985755    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:08.985892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985780    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:08.985892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.985793    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:08.987580 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.987549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:08.987580 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.987570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm" (OuterVolumeSpecName: "dshm") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:08.987790 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:08.987652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98" (OuterVolumeSpecName: "kube-api-access-fwb98") pod "bb619896-252e-4ebf-8c0b-dfabf6782eed" (UID: "bb619896-252e-4ebf-8c0b-dfabf6782eed"). InnerVolumeSpecName "kube-api-access-fwb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:09.086492 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.086461    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb619896-252e-4ebf-8c0b-dfabf6782eed-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.086492 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.086485    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb619896-252e-4ebf-8c0b-dfabf6782eed-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.086492 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.086497    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwb98\" (UniqueName: \"kubernetes.io/projected/bb619896-252e-4ebf-8c0b-dfabf6782eed-kube-api-access-fwb98\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.609657 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.609618    2581 generic.go:356] "Generic (PLEG): container finished" podID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerID="90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25" exitCode=0
Sep 17 19:23:09.610072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.609668    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"
Sep 17 19:23:09.610072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.609667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" event={"ID":"bb619896-252e-4ebf-8c0b-dfabf6782eed","Type":"ContainerDied","Data":"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"}
Sep 17 19:23:09.610072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.609781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g" event={"ID":"bb619896-252e-4ebf-8c0b-dfabf6782eed","Type":"ContainerDied","Data":"fa3d6d4a73d425b210901eb4bdf54e4c413319bc117cdebd27bdffc01ad27231"}
Sep 17 19:23:09.610072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.609810    2581 scope.go:117] "RemoveContainer" containerID="90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"
Sep 17 19:23:09.620101 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.620082    2581 scope.go:117] "RemoveContainer" containerID="90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"
Sep 17 19:23:09.620391 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:23:09.620372    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25\": container with ID starting with 90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25 not found: ID does not exist" containerID="90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"
Sep 17 19:23:09.620495 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.620398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25"} err="failed to get container status \"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25\": rpc error: code = NotFound desc = could not find container \"90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25\": container with ID starting with 90c5d037feac47ed7667364028b078f8f4dcb5009e2ac6ef6740ba84c4dc8b25 not found: ID does not exist"
Sep 17 19:23:09.631539 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.631509    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:23:09.636020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:09.635992    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-5z59g"]
Sep 17 19:23:10.138174 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:10.138145    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" path="/var/lib/kubelet/pods/bb619896-252e-4ebf-8c0b-dfabf6782eed/volumes"
Sep 17 19:23:11.342617 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:11.342581    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:16.169935 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.169901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main"
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main"
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170285    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerName="main"
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170291    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerName="main"
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170358    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47284ffb-ae9e-462a-9a91-bfea7bcdb38c" containerName="main"
Sep 17 19:23:16.170443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.170365    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb619896-252e-4ebf-8c0b-dfabf6782eed" containerName="main"
Sep 17 19:23:16.174851 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.174828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.178169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.178144    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 19:23:16.178355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.178150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-l82t5\""
Sep 17 19:23:16.178509 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.178218    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 19:23:16.178580 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.178283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-w6txw\""
Sep 17 19:23:16.178722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.178284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 19:23:16.186458 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.186432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:23:16.246737 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.246705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxrf\" (UniqueName: \"kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.246737 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.246751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.347170 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.347137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.347342 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.347231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxrf\" (UniqueName: \"kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.349766 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.349740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.354694 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.354675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxrf\" (UniqueName: \"kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.488843 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.488739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:16.765206 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:16.765126    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:23:16.768522 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:23:16.768487    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd418683a_ef95_4adf_826d_ed0fb853fc57.slice/crio-14d9e19f319310d73cba7ab7024533f485c883e76d4dc357793261914042b743 WatchSource:0}: Error finding container 14d9e19f319310d73cba7ab7024533f485c883e76d4dc357793261914042b743: Status 404 returned error can't find the container with id 14d9e19f319310d73cba7ab7024533f485c883e76d4dc357793261914042b743
Sep 17 19:23:17.654262 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:17.654226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" event={"ID":"d418683a-ef95-4adf-826d-ed0fb853fc57","Type":"ContainerStarted","Data":"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"}
Sep 17 19:23:17.654648 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:17.654278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" event={"ID":"d418683a-ef95-4adf-826d-ed0fb853fc57","Type":"ContainerStarted","Data":"14d9e19f319310d73cba7ab7024533f485c883e76d4dc357793261914042b743"}
Sep 17 19:23:17.654648 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:17.654356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:17.672608 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:17.672555    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podStartSLOduration=1.672536746 podStartE2EDuration="1.672536746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:17.670887757 +0000 UTC m=+1029.684457919" watchObservedRunningTime="2026-09-17 19:23:17.672536746 +0000 UTC m=+1029.686106906"
Sep 17 19:23:21.341991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:21.341941    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:23.615664 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:23.615619    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" containerID="cri-o://d08e37291c011a5901cf20397b6f1d453a1be1b77d7b08e3a0329ebdca5242d1" gracePeriod=45
Sep 17 19:23:31.342362 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:31.342321    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:31.342785 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:31.342458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:23:41.342890 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:41.342849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:48.660609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:48.660579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:23:51.342455 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:51.342350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:53.822315 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.822284    2581 generic.go:356] "Generic (PLEG): container finished" podID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerID="d08e37291c011a5901cf20397b6f1d453a1be1b77d7b08e3a0329ebdca5242d1" exitCode=0
Sep 17 19:23:53.822721 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.822356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" event={"ID":"efb63de1-f5db-47f2-8928-8a9aaa454b30","Type":"ContainerDied","Data":"d08e37291c011a5901cf20397b6f1d453a1be1b77d7b08e3a0329ebdca5242d1"}
Sep 17 19:23:53.869651 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.869624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:23:53.973943 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.973910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jjnh\" (UniqueName: \"kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh\") pod \"efb63de1-f5db-47f2-8928-8a9aaa454b30\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") "
Sep 17 19:23:53.974179 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.973978    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs\") pod \"efb63de1-f5db-47f2-8928-8a9aaa454b30\" (UID: \"efb63de1-f5db-47f2-8928-8a9aaa454b30\") "
Sep 17 19:23:53.976241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.976204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh" (OuterVolumeSpecName: "kube-api-access-4jjnh") pod "efb63de1-f5db-47f2-8928-8a9aaa454b30" (UID: "efb63de1-f5db-47f2-8928-8a9aaa454b30"). InnerVolumeSpecName "kube-api-access-4jjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:53.976241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:53.976209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb63de1-f5db-47f2-8928-8a9aaa454b30" (UID: "efb63de1-f5db-47f2-8928-8a9aaa454b30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:54.075070 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.075031    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb63de1-f5db-47f2-8928-8a9aaa454b30-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:54.075070 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.075064    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjnh\" (UniqueName: \"kubernetes.io/projected/efb63de1-f5db-47f2-8928-8a9aaa454b30-kube-api-access-4jjnh\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:23:54.828559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.828525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42" event={"ID":"efb63de1-f5db-47f2-8928-8a9aaa454b30","Type":"ContainerDied","Data":"4528007c5ef4f9ef2d593f5835af0b37444e402d4e155073d41fa969f2498c0e"}
Sep 17 19:23:54.829083 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.828585    2581 scope.go:117] "RemoveContainer" containerID="d08e37291c011a5901cf20397b6f1d453a1be1b77d7b08e3a0329ebdca5242d1"
Sep 17 19:23:54.829083 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.828539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"
Sep 17 19:23:54.844902 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.844837    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:23:54.846272 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:54.846252    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-758dq42"]
Sep 17 19:23:56.137752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:23:56.137722    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" path="/var/lib/kubelet/pods/efb63de1-f5db-47f2-8928-8a9aaa454b30/volumes"
Sep 17 19:24:04.009758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.009723    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:04.010125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.010040    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main"
Sep 17 19:24:04.010125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.010050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main"
Sep 17 19:24:04.010125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.010106    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb63de1-f5db-47f2-8928-8a9aaa454b30" containerName="main"
Sep 17 19:24:04.014202 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.014178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.018728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.018684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-spldg\""
Sep 17 19:24:04.018728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.018710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 19:24:04.018920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.018793    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 19:24:04.018920 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.018716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-hptdm\""
Sep 17 19:24:04.019819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.019802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 19:24:04.029573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.029545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:04.048428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.048601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.048601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.048601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65l4\" (UniqueName: \"kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.048601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.048779 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.048692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149057 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149057 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149300 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65l4\" (UniqueName: \"kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149300 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149300 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149300 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149598 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149598 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.149598 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.149583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.151618 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.151598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.151872 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.151855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.157253 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.157233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65l4\" (UniqueName: \"kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.247463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.247429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:24:04.250789 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.250770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.253820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.253801    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-r42fn\""
Sep 17 19:24:04.262386 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.262336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:24:04.326018 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.325979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:04.351271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.351236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.351388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.351309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxbz\" (UniqueName: \"kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.452569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.452540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxbz\" (UniqueName: \"kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.452754 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.452600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.455181 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.455155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.461030 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.461010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxbz\" (UniqueName: \"kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.561442 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.561339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:04.603107 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.603072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:04.606404 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:24:04.606377    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74e8082_edad_4f86_b699_272ce7a3b7f5.slice/crio-b5623d2d57ab8b34ee5fa1e8b663f48aedf7c787868da670de38596c15944c20 WatchSource:0}: Error finding container b5623d2d57ab8b34ee5fa1e8b663f48aedf7c787868da670de38596c15944c20: Status 404 returned error can't find the container with id b5623d2d57ab8b34ee5fa1e8b663f48aedf7c787868da670de38596c15944c20
Sep 17 19:24:04.608587 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.608567    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:24:04.836359 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.836267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:24:04.840289 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:24:04.840260    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b1eb9_5d0d_4e24_9255_c7b74bbef48e.slice/crio-19a5c6009f804ae2d323b9fc5c8e79f6b77ad9cc405ed7a07921aa46824cdd00 WatchSource:0}: Error finding container 19a5c6009f804ae2d323b9fc5c8e79f6b77ad9cc405ed7a07921aa46824cdd00: Status 404 returned error can't find the container with id 19a5c6009f804ae2d323b9fc5c8e79f6b77ad9cc405ed7a07921aa46824cdd00
Sep 17 19:24:04.873803 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.873770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" event={"ID":"322b1eb9-5d0d-4e24-9255-c7b74bbef48e","Type":"ContainerStarted","Data":"19a5c6009f804ae2d323b9fc5c8e79f6b77ad9cc405ed7a07921aa46824cdd00"}
Sep 17 19:24:04.875240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:04.875206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerStarted","Data":"b5623d2d57ab8b34ee5fa1e8b663f48aedf7c787868da670de38596c15944c20"}
Sep 17 19:24:05.887431 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:05.887360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" event={"ID":"322b1eb9-5d0d-4e24-9255-c7b74bbef48e","Type":"ContainerStarted","Data":"4e52ba8528aaece90b51a5af86c19c87ff6c70511fac7540db5b9ad54655261c"}
Sep 17 19:24:05.887832 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:05.887531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:05.906123 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:05.906072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podStartSLOduration=1.906056612 podStartE2EDuration="1.906056612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:05.903482399 +0000 UTC m=+1077.917052560" watchObservedRunningTime="2026-09-17 19:24:05.906056612 +0000 UTC m=+1077.919626772"
Sep 17 19:24:06.893596 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:06.893560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerStarted","Data":"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"}
Sep 17 19:24:06.894057 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:06.893706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:07.900470 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:07.900401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerStarted","Data":"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"}
Sep 17 19:24:07.937144 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:07.937068    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" podStartSLOduration=3.311305744 podStartE2EDuration="4.937048407s" totalImagesPullingTime="1.625742663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:07.936920597 +0000 UTC m=+1079.950490759" watchObservedRunningTime="2026-09-17 19:24:07.937048407 +0000 UTC m=+1079.950618568"
Sep 17 19:24:08.326648 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:08.326595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:14.327033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:14.326990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:14.339438 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:14.339402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:14.942100 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:14.942071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:25.950507 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:25.950474    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:36.895865 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:36.895832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:24:39.030365 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.030333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:39.034997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.034966    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:24:39.042352 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:24:39.042305    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:24:39.042352 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.042352 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.042352 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a" cmd=["/bin/sleep","15"]
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.042372    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" message="" err=<
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:24:39.042393    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="main" containerID="cri-o://58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"
Sep 17 19:24:39.042736 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.042463    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="main" containerID="cri-o://58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a" gracePeriod=60
Sep 17 19:24:39.260631 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.260360    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="llm-d-routing-sidecar" containerID="cri-o://d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967" gracePeriod=60
Sep 17 19:24:39.504463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.504439    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:39.656944 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.656846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.656944 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.656891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.656944 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.656946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.657220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.656975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.657220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w65l4\" (UniqueName: \"kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.657220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home\") pod \"c74e8082-edad-4f86-b699-272ce7a3b7f5\" (UID: \"c74e8082-edad-4f86-b699-272ce7a3b7f5\") "
Sep 17 19:24:39.657220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.657391 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657341    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.657458 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657382    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.657458 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.657390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home" (OuterVolumeSpecName: "home") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.659531 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.659505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4" (OuterVolumeSpecName: "kube-api-access-w65l4") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "kube-api-access-w65l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:39.659531 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.659509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:39.659677 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.659509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm" (OuterVolumeSpecName: "dshm") pod "c74e8082-edad-4f86-b699-272ce7a3b7f5" (UID: "c74e8082-edad-4f86-b699-272ce7a3b7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.758007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.757953    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74e8082-edad-4f86-b699-272ce7a3b7f5-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.758007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.757999    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.758007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.758008    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65l4\" (UniqueName: \"kubernetes.io/projected/c74e8082-edad-4f86-b699-272ce7a3b7f5-kube-api-access-w65l4\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.758007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.758017    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.758007 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:39.758025    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74e8082-edad-4f86-b699-272ce7a3b7f5-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:24:40.050581 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050551    2581 generic.go:356] "Generic (PLEG): container finished" podID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a" exitCode=0
Sep 17 19:24:40.050581 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050577    2581 generic.go:356] "Generic (PLEG): container finished" podID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerID="d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967" exitCode=0
Sep 17 19:24:40.051046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050598    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"
Sep 17 19:24:40.051046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerDied","Data":"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"}
Sep 17 19:24:40.051046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerDied","Data":"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"}
Sep 17 19:24:40.051046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8" event={"ID":"c74e8082-edad-4f86-b699-272ce7a3b7f5","Type":"ContainerDied","Data":"b5623d2d57ab8b34ee5fa1e8b663f48aedf7c787868da670de38596c15944c20"}
Sep 17 19:24:40.051046 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.050722    2581 scope.go:117] "RemoveContainer" containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"
Sep 17 19:24:40.059841 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.059822    2581 scope.go:117] "RemoveContainer" containerID="d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"
Sep 17 19:24:40.068188 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.068170    2581 scope.go:117] "RemoveContainer" containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"
Sep 17 19:24:40.068480 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:24:40.068462    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a\": container with ID starting with 58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a not found: ID does not exist" containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"
Sep 17 19:24:40.068531 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.068490    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"} err="failed to get container status \"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a\": rpc error: code = NotFound desc = could not find container \"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a\": container with ID starting with 58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a not found: ID does not exist"
Sep 17 19:24:40.068531 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.068507    2581 scope.go:117] "RemoveContainer" containerID="d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"
Sep 17 19:24:40.068773 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:24:40.068755    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967\": container with ID starting with d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967 not found: ID does not exist" containerID="d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"
Sep 17 19:24:40.068826 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.068776    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"} err="failed to get container status \"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967\": rpc error: code = NotFound desc = could not find container \"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967\": container with ID starting with d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967 not found: ID does not exist"
Sep 17 19:24:40.068826 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.068790    2581 scope.go:117] "RemoveContainer" containerID="58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"
Sep 17 19:24:40.069037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.069015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a"} err="failed to get container status \"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a\": rpc error: code = NotFound desc = could not find container \"58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a\": container with ID starting with 58fc3d0cc1ea67daa722914fe2864b7346a1a38075085f648fe417d4cd25b26a not found: ID does not exist"
Sep 17 19:24:40.069077 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.069038    2581 scope.go:117] "RemoveContainer" containerID="d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"
Sep 17 19:24:40.069272 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.069255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967"} err="failed to get container status \"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967\": rpc error: code = NotFound desc = could not find container \"d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967\": container with ID starting with d79da030d2955cbffd270f0d872ac89a416c8f95fcdcc65dc0eeaadd5a0d2967 not found: ID does not exist"
Sep 17 19:24:40.072575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.072550    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:40.077224 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.077199    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-47st8"]
Sep 17 19:24:40.137396 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:40.137366    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" path="/var/lib/kubelet/pods/c74e8082-edad-4f86-b699-272ce7a3b7f5/volumes"
Sep 17 19:24:46.895484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:46.895447    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:24:54.056500 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:54.056452    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main" containerID="cri-o://4e52ba8528aaece90b51a5af86c19c87ff6c70511fac7540db5b9ad54655261c" gracePeriod=45
Sep 17 19:24:56.896318 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:24:56.896280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:25:06.896034 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:06.895994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:25:06.896476 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:06.896115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:25:16.895295 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:16.895259    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:25:24.257814 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.257777    2581 generic.go:356] "Generic (PLEG): container finished" podID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerID="4e52ba8528aaece90b51a5af86c19c87ff6c70511fac7540db5b9ad54655261c" exitCode=0
Sep 17 19:25:24.258144 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.257844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" event={"ID":"322b1eb9-5d0d-4e24-9255-c7b74bbef48e","Type":"ContainerDied","Data":"4e52ba8528aaece90b51a5af86c19c87ff6c70511fac7540db5b9ad54655261c"}
Sep 17 19:25:24.307178 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.307155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:25:24.425542 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.425516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs\") pod \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") "
Sep 17 19:25:24.425704 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.425592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klxbz\" (UniqueName: \"kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz\") pod \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\" (UID: \"322b1eb9-5d0d-4e24-9255-c7b74bbef48e\") "
Sep 17 19:25:24.427964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.427925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz" (OuterVolumeSpecName: "kube-api-access-klxbz") pod "322b1eb9-5d0d-4e24-9255-c7b74bbef48e" (UID: "322b1eb9-5d0d-4e24-9255-c7b74bbef48e"). InnerVolumeSpecName "kube-api-access-klxbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:25:24.427964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.427929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "322b1eb9-5d0d-4e24-9255-c7b74bbef48e" (UID: "322b1eb9-5d0d-4e24-9255-c7b74bbef48e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:25:24.526756 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.526719    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:25:24.526756 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:24.526750    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klxbz\" (UniqueName: \"kubernetes.io/projected/322b1eb9-5d0d-4e24-9255-c7b74bbef48e-kube-api-access-klxbz\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:25:25.263479 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:25.263446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs" event={"ID":"322b1eb9-5d0d-4e24-9255-c7b74bbef48e","Type":"ContainerDied","Data":"19a5c6009f804ae2d323b9fc5c8e79f6b77ad9cc405ed7a07921aa46824cdd00"}
Sep 17 19:25:25.263479 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:25.263467    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"
Sep 17 19:25:25.263978 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:25.263513    2581 scope.go:117] "RemoveContainer" containerID="4e52ba8528aaece90b51a5af86c19c87ff6c70511fac7540db5b9ad54655261c"
Sep 17 19:25:25.284594 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:25.284569    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:25:25.290439 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:25.290399    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedd8fqs"]
Sep 17 19:25:26.137520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:26.137491    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" path="/var/lib/kubelet/pods/322b1eb9-5d0d-4e24-9255-c7b74bbef48e/volumes"
Sep 17 19:25:37.288093 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="llm-d-routing-sidecar"
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288609    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="llm-d-routing-sidecar"
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="main"
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="main"
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288672    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main"
Sep 17 19:25:37.288715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288682    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main"
Sep 17 19:25:37.289029 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="llm-d-routing-sidecar"
Sep 17 19:25:37.289029 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288786    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="322b1eb9-5d0d-4e24-9255-c7b74bbef48e" containerName="main"
Sep 17 19:25:37.289029 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.288797    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74e8082-edad-4f86-b699-272ce7a3b7f5" containerName="main"
Sep 17 19:25:37.292043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.292023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.294302 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.294276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:25:37.294566 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.294545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-wrwx8\""
Sep 17 19:25:37.294625 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.294545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 19:25:37.295201 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.295182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 19:25:37.295296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.295189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 19:25:37.295296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.295225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 19:25:37.300979 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.300958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:25:37.435217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9tq\" (UniqueName: \"kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.435612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.435480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536689 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536689 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536689 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9tq\" (UniqueName: \"kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536689 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536901 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536901 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.536901 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.537050 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.536953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.537165 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.537135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.537570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.537511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.537770 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.537743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.538168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.537946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.539338 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.539318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.540008 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.539990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.546591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.546557    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:25:37.550331 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.550312    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.552179 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.552146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9tq\" (UniqueName: \"kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.552606 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.552587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-vgdjv\""
Sep 17 19:25:37.562776 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.562752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:25:37.603904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.603876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:37.637731 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.637696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqxh\" (UniqueName: \"kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.637902 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.637748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.739141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.739097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqxh\" (UniqueName: \"kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.739141 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.739139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.742111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.742079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.750614 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.750592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqxh\" (UniqueName: \"kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.882165 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.882078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:37.884357 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:37.884288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:25:37.886915 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:25:37.886884    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7489a631_58b6_495c_9ed7_e03873dc8fd5.slice/crio-1d45f97942af999a2d8b20bbb4b29eecf34e0622bd17ad177bf0498df51c577f WatchSource:0}: Error finding container 1d45f97942af999a2d8b20bbb4b29eecf34e0622bd17ad177bf0498df51c577f: Status 404 returned error can't find the container with id 1d45f97942af999a2d8b20bbb4b29eecf34e0622bd17ad177bf0498df51c577f
Sep 17 19:25:38.156760 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.156734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:25:38.159402 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:25:38.159372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96b058a_f454_4ca4_b574_5c9bdb793b9e.slice/crio-986ab49e568b9212b6ab3d9064d1affac6d27ae81da90c3dbe8b03a8d353493c WatchSource:0}: Error finding container 986ab49e568b9212b6ab3d9064d1affac6d27ae81da90c3dbe8b03a8d353493c: Status 404 returned error can't find the container with id 986ab49e568b9212b6ab3d9064d1affac6d27ae81da90c3dbe8b03a8d353493c
Sep 17 19:25:38.328409 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.328366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerStarted","Data":"1d45f97942af999a2d8b20bbb4b29eecf34e0622bd17ad177bf0498df51c577f"}
Sep 17 19:25:38.331685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.331155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" event={"ID":"d96b058a-f454-4ca4-b574-5c9bdb793b9e","Type":"ContainerStarted","Data":"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"}
Sep 17 19:25:38.331685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.331205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" event={"ID":"d96b058a-f454-4ca4-b574-5c9bdb793b9e","Type":"ContainerStarted","Data":"986ab49e568b9212b6ab3d9064d1affac6d27ae81da90c3dbe8b03a8d353493c"}
Sep 17 19:25:38.331685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.331311    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:25:38.349789 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:38.349732    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podStartSLOduration=1.349711722 podStartE2EDuration="1.349711722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:38.349285418 +0000 UTC m=+1170.362855574" watchObservedRunningTime="2026-09-17 19:25:38.349711722 +0000 UTC m=+1170.363281911"
Sep 17 19:25:43.363013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:43.362964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerStarted","Data":"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"}
Sep 17 19:25:45.373571 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:45.373536    2581 generic.go:356] "Generic (PLEG): container finished" podID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerID="44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a" exitCode=0
Sep 17 19:25:45.374030 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:45.373618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerDied","Data":"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"}
Sep 17 19:25:46.380696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:46.380659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerStarted","Data":"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"}
Sep 17 19:25:46.400308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:46.400249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" podStartSLOduration=2.615744562 podStartE2EDuration="9.400231713s" totalImagesPullingTime="4.58722238s" totalInitContainerRuntime="2.197264771s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:46.39995081 +0000 UTC m=+1178.413520971" watchObservedRunningTime="2026-09-17 19:25:46.400231713 +0000 UTC m=+1178.413801876"
Sep 17 19:25:46.604824 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:46.604786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:47.604310 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:47.604279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:47.617033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:47.617009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:25:48.402095 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:25:48.402063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:26:09.339913 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:26:09.339883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:26:10.153075 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:26:10.153040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:26:10.154970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:26:10.154944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:27:06.624052 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:06.623960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:27:08.646886 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:08.646838    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:27:08.648615 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:08.648574    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:27:08.657018 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:08.656982    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:27:08.657018 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.657018 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.657018 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb" cmd=["/bin/sleep","15"]
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:08.657037    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" message="" err=<
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:08.657052    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="main" containerID="cri-o://d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"
Sep 17 19:27:08.657290 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:08.657106    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="main" containerID="cri-o://d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb" gracePeriod=60
Sep 17 19:27:08.660349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:08.660321    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:09.004856 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.004832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:27:09.025886 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj9tq\" (UniqueName: \"kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025929    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.025999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026045    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm\") pod \"7489a631-58b6-495c-9ed7-e03873dc8fd5\" (UID: \"7489a631-58b6-495c-9ed7-e03873dc8fd5\") "
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home" (OuterVolumeSpecName: "home") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026330    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.026536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.026373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:09.029316 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.028967    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm" (OuterVolumeSpecName: "dshm") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.029620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.029593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq" (OuterVolumeSpecName: "kube-api-access-wj9tq") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "kube-api-access-wj9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:09.029986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.029951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:09.053941 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.053887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7489a631-58b6-495c-9ed7-e03873dc8fd5" (UID: "7489a631-58b6-495c-9ed7-e03873dc8fd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.127409 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127377    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127409 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127402    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127412    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127448    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj9tq\" (UniqueName: \"kubernetes.io/projected/7489a631-58b6-495c-9ed7-e03873dc8fd5-kube-api-access-wj9tq\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127457    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7489a631-58b6-495c-9ed7-e03873dc8fd5-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127465    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7489a631-58b6-495c-9ed7-e03873dc8fd5-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.127621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.127472    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7489a631-58b6-495c-9ed7-e03873dc8fd5-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.340750 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.340645    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:09.772680 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.772646    2581 generic.go:356] "Generic (PLEG): container finished" podID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerID="d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb" exitCode=0
Sep 17 19:27:09.773078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.772706    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"
Sep 17 19:27:09.773078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.772734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerDied","Data":"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"}
Sep 17 19:27:09.773078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.772809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h" event={"ID":"7489a631-58b6-495c-9ed7-e03873dc8fd5","Type":"ContainerDied","Data":"1d45f97942af999a2d8b20bbb4b29eecf34e0622bd17ad177bf0498df51c577f"}
Sep 17 19:27:09.773078 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.772831    2581 scope.go:117] "RemoveContainer" containerID="d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"
Sep 17 19:27:09.782118 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.782098    2581 scope.go:117] "RemoveContainer" containerID="44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"
Sep 17 19:27:09.789720 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.789700    2581 scope.go:117] "RemoveContainer" containerID="d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"
Sep 17 19:27:09.790006 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:09.789986    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb\": container with ID starting with d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb not found: ID does not exist" containerID="d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"
Sep 17 19:27:09.790092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.790014    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb"} err="failed to get container status \"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb\": rpc error: code = NotFound desc = could not find container \"d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb\": container with ID starting with d58fcaf2debb0dd29be7acec05e255b006d6fc3e707561937d65baa56f338edb not found: ID does not exist"
Sep 17 19:27:09.790092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.790032    2581 scope.go:117] "RemoveContainer" containerID="44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"
Sep 17 19:27:09.790311 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:09.790293    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a\": container with ID starting with 44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a not found: ID does not exist" containerID="44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"
Sep 17 19:27:09.790357 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.790316    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a"} err="failed to get container status \"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a\": rpc error: code = NotFound desc = could not find container \"44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a\": container with ID starting with 44f043b8ec79b79369f90685209966a74463b83464a78b145f0a4ebac0e5d62a not found: ID does not exist"
Sep 17 19:27:09.796794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.796774    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:27:09.801553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:09.801532    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-5p42h"]
Sep 17 19:27:10.137970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:10.137892    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" path="/var/lib/kubelet/pods/7489a631-58b6-495c-9ed7-e03873dc8fd5/volumes"
Sep 17 19:27:18.659986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:18.659945    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:19.340607 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:19.340570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:21.644331 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:21.644282    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" containerID="cri-o://670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9" gracePeriod=45
Sep 17 19:27:23.668711 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:23.668665    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" containerID="cri-o://98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f" gracePeriod=45
Sep 17 19:27:28.660617 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:28.660574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:28.660988 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:28.660676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:27:29.339964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:29.339922    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:29.340151 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:29.340032    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:27:38.659890 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:38.659855    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:39.340758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:39.340713    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:48.660477 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:48.660442    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:49.340216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:49.340172    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:51.899825 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.899792    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:27:51.961168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.961131    2581 generic.go:356] "Generic (PLEG): container finished" podID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerID="670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9" exitCode=0
Sep 17 19:27:51.961349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.961193    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"
Sep 17 19:27:51.961349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.961217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" event={"ID":"d418683a-ef95-4adf-826d-ed0fb853fc57","Type":"ContainerDied","Data":"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"}
Sep 17 19:27:51.961349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.961302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz" event={"ID":"d418683a-ef95-4adf-826d-ed0fb853fc57","Type":"ContainerDied","Data":"14d9e19f319310d73cba7ab7024533f485c883e76d4dc357793261914042b743"}
Sep 17 19:27:51.961349 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.961326    2581 scope.go:117] "RemoveContainer" containerID="670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"
Sep 17 19:27:51.969883 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.969866    2581 scope.go:117] "RemoveContainer" containerID="670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"
Sep 17 19:27:51.970151 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:51.970134    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9\": container with ID starting with 670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9 not found: ID does not exist" containerID="670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"
Sep 17 19:27:51.970212 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:51.970159    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9"} err="failed to get container status \"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9\": rpc error: code = NotFound desc = could not find container \"670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9\": container with ID starting with 670d6964e25c28c38ce27a9bb830eff559fe8dee4ef871d0bb7fa9ea2bc51dd9 not found: ID does not exist"
Sep 17 19:27:52.085440 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.085378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmxrf\" (UniqueName: \"kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf\") pod \"d418683a-ef95-4adf-826d-ed0fb853fc57\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") "
Sep 17 19:27:52.085601 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.085515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs\") pod \"d418683a-ef95-4adf-826d-ed0fb853fc57\" (UID: \"d418683a-ef95-4adf-826d-ed0fb853fc57\") "
Sep 17 19:27:52.087814 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.087779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf" (OuterVolumeSpecName: "kube-api-access-fmxrf") pod "d418683a-ef95-4adf-826d-ed0fb853fc57" (UID: "d418683a-ef95-4adf-826d-ed0fb853fc57"). InnerVolumeSpecName "kube-api-access-fmxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:52.087942 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.087787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d418683a-ef95-4adf-826d-ed0fb853fc57" (UID: "d418683a-ef95-4adf-826d-ed0fb853fc57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:52.186124 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.186089    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d418683a-ef95-4adf-826d-ed0fb853fc57-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:52.186124 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.186120    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxrf\" (UniqueName: \"kubernetes.io/projected/d418683a-ef95-4adf-826d-ed0fb853fc57-kube-api-access-fmxrf\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:52.277256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.277178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:27:52.280953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:52.280930    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-kbfrz"]
Sep 17 19:27:53.921483 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.921460    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:27:53.971864 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.971790    2581 generic.go:356] "Generic (PLEG): container finished" podID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerID="98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f" exitCode=0
Sep 17 19:27:53.971864 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.971835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"
Sep 17 19:27:53.971864 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.971849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" event={"ID":"d96b058a-f454-4ca4-b574-5c9bdb793b9e","Type":"ContainerDied","Data":"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"}
Sep 17 19:27:53.972091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.971890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr" event={"ID":"d96b058a-f454-4ca4-b574-5c9bdb793b9e","Type":"ContainerDied","Data":"986ab49e568b9212b6ab3d9064d1affac6d27ae81da90c3dbe8b03a8d353493c"}
Sep 17 19:27:53.972091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.971911    2581 scope.go:117] "RemoveContainer" containerID="98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"
Sep 17 19:27:53.980239 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.980223    2581 scope.go:117] "RemoveContainer" containerID="98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"
Sep 17 19:27:53.980564 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:27:53.980544    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f\": container with ID starting with 98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f not found: ID does not exist" containerID="98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"
Sep 17 19:27:53.980641 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:53.980576    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f"} err="failed to get container status \"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f\": rpc error: code = NotFound desc = could not find container \"98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f\": container with ID starting with 98afb700fcf607ea5a7c2f12dcd4313ed32ac13c807784d71bca9458515d689f not found: ID does not exist"
Sep 17 19:27:54.102728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.102689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gqxh\" (UniqueName: \"kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh\") pod \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") "
Sep 17 19:27:54.102926 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.102748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs\") pod \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\" (UID: \"d96b058a-f454-4ca4-b574-5c9bdb793b9e\") "
Sep 17 19:27:54.105071 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.105044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d96b058a-f454-4ca4-b574-5c9bdb793b9e" (UID: "d96b058a-f454-4ca4-b574-5c9bdb793b9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:54.105071 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.105047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh" (OuterVolumeSpecName: "kube-api-access-2gqxh") pod "d96b058a-f454-4ca4-b574-5c9bdb793b9e" (UID: "d96b058a-f454-4ca4-b574-5c9bdb793b9e"). InnerVolumeSpecName "kube-api-access-2gqxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:54.136560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.136533    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" path="/var/lib/kubelet/pods/d418683a-ef95-4adf-826d-ed0fb853fc57/volumes"
Sep 17 19:27:54.203986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.203953    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqxh\" (UniqueName: \"kubernetes.io/projected/d96b058a-f454-4ca4-b574-5c9bdb793b9e-kube-api-access-2gqxh\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:54.203986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.203979    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96b058a-f454-4ca4-b574-5c9bdb793b9e-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:27:54.287925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.287857    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:27:54.291806 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:54.291776    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69vp5wr"]
Sep 17 19:27:56.136971 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:27:56.136941    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" path="/var/lib/kubelet/pods/d96b058a-f454-4ca4-b574-5c9bdb793b9e/volumes"
Sep 17 19:28:10.388748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.388716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="storage-initializer"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389049    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="storage-initializer"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389065    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389083    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389105    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389158    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96b058a-f454-4ca4-b574-5c9bdb793b9e" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389167    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7489a631-58b6-495c-9ed7-e03873dc8fd5" containerName="main"
Sep 17 19:28:10.389187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.389179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d418683a-ef95-4adf-826d-ed0fb853fc57" containerName="main"
Sep 17 19:28:10.393367 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.393346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.397044 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.397017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-rw76k\""
Sep 17 19:28:10.397044 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.397031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 19:28:10.397256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.397023    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 19:28:10.397256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.397050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 19:28:10.397256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.397081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xrhg4\""
Sep 17 19:28:10.404230 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.404208    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:28:10.423525 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.423494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.423662 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.423553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725pr\" (UniqueName: \"kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.525022 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.524925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.525022 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.525012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725pr\" (UniqueName: \"kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.527931 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.527905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.535576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.535553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-725pr\" (UniqueName: \"kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.705211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.705174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:10.978880 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:10.978850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:28:10.982121 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:28:10.982064    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5954b1_528e_4e6c_b9f9_b7964183d394.slice/crio-af865732048104e33c71892526448bad4e380fdfc93f4adf9ffe70753a16d08b WatchSource:0}: Error finding container af865732048104e33c71892526448bad4e380fdfc93f4adf9ffe70753a16d08b: Status 404 returned error can't find the container with id af865732048104e33c71892526448bad4e380fdfc93f4adf9ffe70753a16d08b
Sep 17 19:28:11.049867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:11.049837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" event={"ID":"ff5954b1-528e-4e6c-b9f9-b7964183d394","Type":"ContainerStarted","Data":"af865732048104e33c71892526448bad4e380fdfc93f4adf9ffe70753a16d08b"}
Sep 17 19:28:12.055850 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:12.055812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" event={"ID":"ff5954b1-528e-4e6c-b9f9-b7964183d394","Type":"ContainerStarted","Data":"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"}
Sep 17 19:28:12.056206 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:12.055883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:12.074636 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:12.074586    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podStartSLOduration=2.074570896 podStartE2EDuration="2.074570896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:12.072880969 +0000 UTC m=+1324.086451132" watchObservedRunningTime="2026-09-17 19:28:12.074570896 +0000 UTC m=+1324.088141056"
Sep 17 19:28:43.062955 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.062878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:28:43.965244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.965210    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:28:43.969072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.969049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:43.972021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.971987    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:28:43.972021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.972010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9rs56\""
Sep 17 19:28:43.972184 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.972027    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 19:28:43.972521 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.972508    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 19:28:43.972755 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.972741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 19:28:43.973098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.973077    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 19:28:43.980080 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:43.980057    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:28:44.100794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.100794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.101218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.100941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9j2\" (UniqueName: \"kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.201947 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.201906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.201947 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.201948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.201966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.201995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9j2\" (UniqueName: \"kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202400 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202400 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202549 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202598 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.202702 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.202686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.203191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.203168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.203231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.203196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.204562 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.204541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.205066 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.205045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.216081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.216016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9j2\" (UniqueName: \"kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-t9s9t\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.262162 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.262126    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:28:44.265704 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.265681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.268258 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.268235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-flfk6\""
Sep 17 19:28:44.276709 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.276683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:28:44.281241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.281219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:44.403888 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.403851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.404074 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.403966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hcb\" (UniqueName: \"kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.504733 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.504650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.504862 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.504757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hcb\" (UniqueName: \"kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.507493 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.507463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.513555 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.513536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hcb\" (UniqueName: \"kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-578556x7s\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.555257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.555071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:28:44.557536 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:28:44.557506    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c19e058_5de8_4a9a_9c1b_08bc686f78f3.slice/crio-f0fce728e7412bca7be58ecd95195a9490dfa88897cee55a596d17e127c75650 WatchSource:0}: Error finding container f0fce728e7412bca7be58ecd95195a9490dfa88897cee55a596d17e127c75650: Status 404 returned error can't find the container with id f0fce728e7412bca7be58ecd95195a9490dfa88897cee55a596d17e127c75650
Sep 17 19:28:44.577462 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.577439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:44.853370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:44.853344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:28:44.855580 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:28:44.855547    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b1c035_eb51_4817_85d5_d685b37604c1.slice/crio-f040ab025b7b0a2f05c94f546d508e3e66fd9dbab9d1867c39980347f54bd901 WatchSource:0}: Error finding container f040ab025b7b0a2f05c94f546d508e3e66fd9dbab9d1867c39980347f54bd901: Status 404 returned error can't find the container with id f040ab025b7b0a2f05c94f546d508e3e66fd9dbab9d1867c39980347f54bd901
Sep 17 19:28:45.200829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.200791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" event={"ID":"f8b1c035-eb51-4817-85d5-d685b37604c1","Type":"ContainerStarted","Data":"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"}
Sep 17 19:28:45.201247 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.200863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:28:45.201247 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.200890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" event={"ID":"f8b1c035-eb51-4817-85d5-d685b37604c1","Type":"ContainerStarted","Data":"f040ab025b7b0a2f05c94f546d508e3e66fd9dbab9d1867c39980347f54bd901"}
Sep 17 19:28:45.202396 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.202369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerStarted","Data":"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"}
Sep 17 19:28:45.202515 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.202409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerStarted","Data":"f0fce728e7412bca7be58ecd95195a9490dfa88897cee55a596d17e127c75650"}
Sep 17 19:28:45.221355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:45.221296    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podStartSLOduration=1.2212751 podStartE2EDuration="1.2212751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:45.219031558 +0000 UTC m=+1357.232601720" watchObservedRunningTime="2026-09-17 19:28:45.2212751 +0000 UTC m=+1357.234845262"
Sep 17 19:28:47.214856 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:47.214816    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerID="3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778" exitCode=0
Sep 17 19:28:47.215328 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:47.214889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerDied","Data":"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"}
Sep 17 19:28:48.222200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:48.222154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerStarted","Data":"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"}
Sep 17 19:28:48.243522 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:48.243469    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" podStartSLOduration=2.906757345 podStartE2EDuration="5.243452467s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.336695122s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:48.24279802 +0000 UTC m=+1360.256368182" watchObservedRunningTime="2026-09-17 19:28:48.243452467 +0000 UTC m=+1360.257022627"
Sep 17 19:28:48.281980 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:48.281953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:54.282284 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:54.282249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:54.294640 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:54.294611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:28:55.270112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:28:55.270084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:29:16.209127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:16.209096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:29:21.494643 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.494607    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:29:21.502411 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.502379    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:29:21.511282 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:29:21.511242    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:29:21.511282 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.511282 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.511282 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560" cmd=["/bin/sleep","15"]
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.511308    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" message="" err=<
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:29:21.511324    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="main" containerID="cri-o://1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"
Sep 17 19:29:21.511586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.511395    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="main" containerID="cri-o://1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560" gracePeriod=60
Sep 17 19:29:21.884960 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.884934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:29:21.917629 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.917819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.917819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.917819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb9j2\" (UniqueName: \"kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.917819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.917819 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917795    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.918111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917829    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.918111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home" (OuterVolumeSpecName: "home") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:21.918111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.917889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir\") pod \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\" (UID: \"3c19e058-5de8-4a9a-9c1b-08bc686f78f3\") "
Sep 17 19:29:21.918280 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.918238    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:21.919471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.918361    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:21.919471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.918372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:21.919471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.918401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:21.920634 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.920603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2" (OuterVolumeSpecName: "kube-api-access-hb9j2") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "kube-api-access-hb9j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:21.920796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.920776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm" (OuterVolumeSpecName: "dshm") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:21.920925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.920904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:21.951025 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:21.950973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c19e058-5de8-4a9a-9c1b-08bc686f78f3" (UID: "3c19e058-5de8-4a9a-9c1b-08bc686f78f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.019061 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019027    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019061 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019055    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019061 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019065    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019075    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9j2\" (UniqueName: \"kubernetes.io/projected/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-kube-api-access-hb9j2\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019084    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019091    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.019288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.019098    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c19e058-5de8-4a9a-9c1b-08bc686f78f3-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.380616 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.380575    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerID="1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560" exitCode=0
Sep 17 19:29:22.380802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.380631    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"
Sep 17 19:29:22.380802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.380645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerDied","Data":"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"}
Sep 17 19:29:22.380802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.380699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t" event={"ID":"3c19e058-5de8-4a9a-9c1b-08bc686f78f3","Type":"ContainerDied","Data":"f0fce728e7412bca7be58ecd95195a9490dfa88897cee55a596d17e127c75650"}
Sep 17 19:29:22.380802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.380743    2581 scope.go:117] "RemoveContainer" containerID="1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"
Sep 17 19:29:22.395564 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.395531    2581 scope.go:117] "RemoveContainer" containerID="3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"
Sep 17 19:29:22.400595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.400571    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:29:22.404668 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.404580    2581 scope.go:117] "RemoveContainer" containerID="1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"
Sep 17 19:29:22.405254 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:29:22.404989    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560\": container with ID starting with 1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560 not found: ID does not exist" containerID="1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"
Sep 17 19:29:22.405254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.405031    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560"} err="failed to get container status \"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560\": rpc error: code = NotFound desc = could not find container \"1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560\": container with ID starting with 1ba94c31d0e0bc5580e2837b24c39cd077ff62d55480f8c77a45c29fb6cb9560 not found: ID does not exist"
Sep 17 19:29:22.405254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.405058    2581 scope.go:117] "RemoveContainer" containerID="3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"
Sep 17 19:29:22.405791 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:29:22.405382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778\": container with ID starting with 3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778 not found: ID does not exist" containerID="3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"
Sep 17 19:29:22.405791 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.405403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778"} err="failed to get container status \"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778\": rpc error: code = NotFound desc = could not find container \"3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778\": container with ID starting with 3d171d323723a2fafa220063cfb37be2043ecba2f1458da41de017c1295ef778 not found: ID does not exist"
Sep 17 19:29:22.406664 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:22.406642    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-t9s9t"]
Sep 17 19:29:24.137098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:24.137057    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" path="/var/lib/kubelet/pods/3c19e058-5de8-4a9a-9c1b-08bc686f78f3/volumes"
Sep 17 19:29:26.209137 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:26.209089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:36.209536 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:36.209498    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:36.518285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:36.518188    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" containerID="cri-o://ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667" gracePeriod=45
Sep 17 19:29:46.208785 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:46.208749    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:46.209159 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:46.208856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:29:56.209241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:29:56.209159    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:06.209178 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.209134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:06.773126 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.773102    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:30:06.801392 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.801319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77hcb\" (UniqueName: \"kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb\") pod \"f8b1c035-eb51-4817-85d5-d685b37604c1\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") "
Sep 17 19:30:06.801538 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.801392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs\") pod \"f8b1c035-eb51-4817-85d5-d685b37604c1\" (UID: \"f8b1c035-eb51-4817-85d5-d685b37604c1\") "
Sep 17 19:30:06.803618 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.803585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8b1c035-eb51-4817-85d5-d685b37604c1" (UID: "f8b1c035-eb51-4817-85d5-d685b37604c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:06.803733 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.803665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb" (OuterVolumeSpecName: "kube-api-access-77hcb") pod "f8b1c035-eb51-4817-85d5-d685b37604c1" (UID: "f8b1c035-eb51-4817-85d5-d685b37604c1"). InnerVolumeSpecName "kube-api-access-77hcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:06.902355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.902320    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77hcb\" (UniqueName: \"kubernetes.io/projected/f8b1c035-eb51-4817-85d5-d685b37604c1-kube-api-access-77hcb\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:06.902355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:06.902350    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b1c035-eb51-4817-85d5-d685b37604c1-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:07.590358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.590328    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerID="ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667" exitCode=0
Sep 17 19:30:07.590820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.590372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"
Sep 17 19:30:07.590820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.590406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" event={"ID":"f8b1c035-eb51-4817-85d5-d685b37604c1","Type":"ContainerDied","Data":"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"}
Sep 17 19:30:07.590820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.590475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s" event={"ID":"f8b1c035-eb51-4817-85d5-d685b37604c1","Type":"ContainerDied","Data":"f040ab025b7b0a2f05c94f546d508e3e66fd9dbab9d1867c39980347f54bd901"}
Sep 17 19:30:07.590820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.590497    2581 scope.go:117] "RemoveContainer" containerID="ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"
Sep 17 19:30:07.603253 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.603233    2581 scope.go:117] "RemoveContainer" containerID="ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"
Sep 17 19:30:07.603562 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:30:07.603543    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667\": container with ID starting with ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667 not found: ID does not exist" containerID="ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"
Sep 17 19:30:07.603630 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.603571    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667"} err="failed to get container status \"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667\": rpc error: code = NotFound desc = could not find container \"ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667\": container with ID starting with ed73b1161c8893c11b37d770b9b4c660dd651f7ec56c1bfd7655b2c38ccab667 not found: ID does not exist"
Sep 17 19:30:07.613311 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.613288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:30:07.617341 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:07.617322    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-578556x7s"]
Sep 17 19:30:08.138148 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:08.138117    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" path="/var/lib/kubelet/pods/f8b1c035-eb51-4817-85d5-d685b37604c1/volumes"
Sep 17 19:30:19.691905 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.691868    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692182    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692205    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="storage-initializer"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692211    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="storage-initializer"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692224    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="main"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692229    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="main"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692289    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b1c035-eb51-4817-85d5-d685b37604c1" containerName="main"
Sep 17 19:30:19.692390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.692295    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c19e058-5de8-4a9a-9c1b-08bc686f78f3" containerName="main"
Sep 17 19:30:19.696821 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.696801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.699255 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.699233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 19:30:19.699374 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.699260    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 19:30:19.699457 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.699379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 19:30:19.699646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.699630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:30:19.700217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.700192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 19:30:19.700292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.700198    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-p8278\""
Sep 17 19:30:19.709732 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.709710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:30:19.809382 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnf8\" (UniqueName: \"kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.809717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.809665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910447 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnf8\" (UniqueName: \"kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.910716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.910624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.911197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.911043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.911197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.911109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.911197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.911182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.911480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.911401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.911480 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.911406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.913709 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.913689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.914058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.914035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:19.918639 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:19.918614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnf8\" (UniqueName: \"kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8\") pod \"tokenizer-template-args-test-kserve-8b55465f5-n27f5\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:20.000800 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.000711    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:30:20.005733 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.005711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.006603 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.006578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:20.008392 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.008370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-9bhb5\""
Sep 17 19:30:20.018105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.018080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:30:20.112276 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.112243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnz7\" (UniqueName: \"kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.112444 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.112281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.212938 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.212902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnz7\" (UniqueName: \"kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.212938 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.212944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.215663 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.215635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.221445 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.221398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnz7\" (UniqueName: \"kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.285116 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.285090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:30:20.287838 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:30:20.287814    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b76c37_7916_41ae_9b61_f30a90197cb9.slice/crio-e5d832662eadcbaf7d3d931b9fb104194538e456f7b60d886d3785a57df9d94f WatchSource:0}: Error finding container e5d832662eadcbaf7d3d931b9fb104194538e456f7b60d886d3785a57df9d94f: Status 404 returned error can't find the container with id e5d832662eadcbaf7d3d931b9fb104194538e456f7b60d886d3785a57df9d94f
Sep 17 19:30:20.319106 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.319084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:20.598590 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.598561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:30:20.601279 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:30:20.601255    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193c1b68_e3c5_4d0c_b814_efd93f86abfb.slice/crio-c9237c9a95227f8b1aa2ad5641da0de51395c5a45d0f617546a1cc12c0a396fb WatchSource:0}: Error finding container c9237c9a95227f8b1aa2ad5641da0de51395c5a45d0f617546a1cc12c0a396fb: Status 404 returned error can't find the container with id c9237c9a95227f8b1aa2ad5641da0de51395c5a45d0f617546a1cc12c0a396fb
Sep 17 19:30:20.648812 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.648769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerStarted","Data":"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"}
Sep 17 19:30:20.648974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.648844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerStarted","Data":"e5d832662eadcbaf7d3d931b9fb104194538e456f7b60d886d3785a57df9d94f"}
Sep 17 19:30:20.650327 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:20.650297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" event={"ID":"193c1b68-e3c5-4d0c-b814-efd93f86abfb","Type":"ContainerStarted","Data":"c9237c9a95227f8b1aa2ad5641da0de51395c5a45d0f617546a1cc12c0a396fb"}
Sep 17 19:30:21.664475 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:21.664405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" event={"ID":"193c1b68-e3c5-4d0c-b814-efd93f86abfb","Type":"ContainerStarted","Data":"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"}
Sep 17 19:30:21.664906 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:21.664551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:21.685123 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:21.685058    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podStartSLOduration=2.685036946 podStartE2EDuration="2.685036946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:21.68454878 +0000 UTC m=+1453.698118944" watchObservedRunningTime="2026-09-17 19:30:21.685036946 +0000 UTC m=+1453.698607108"
Sep 17 19:30:22.787370 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:22.787333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:30:23.062966 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:23.062881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:23.675827 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:23.675788    2581 generic.go:356] "Generic (PLEG): container finished" podID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerID="9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c" exitCode=0
Sep 17 19:30:23.675985 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:23.675852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerDied","Data":"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"}
Sep 17 19:30:24.682888 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:24.682849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerStarted","Data":"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"}
Sep 17 19:30:24.705575 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:24.705521    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" podStartSLOduration=3.3558396 podStartE2EDuration="5.705505874s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.349666274s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:24.704063195 +0000 UTC m=+1456.717633357" watchObservedRunningTime="2026-09-17 19:30:24.705505874 +0000 UTC m=+1456.719076034"
Sep 17 19:30:25.007747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:25.007648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:30.007519 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:30.007478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:30.020111 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:30.020083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:30.721643 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:30.721612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:33.062622 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:33.062586    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:37.807069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:37.807023    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" containerID="cri-o://4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc" gracePeriod=45
Sep 17 19:30:43.062148 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:43.062102    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:43.062647 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:43.062226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:30:52.671024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:52.670994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:30:53.062602 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:53.062513    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:57.531283 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.531243    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:30:57.533747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.533716    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:30:57.546471 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:30:57.546430    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:30:57.546471 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.546471 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.546471 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d" cmd=["/bin/sleep","15"]
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.546492    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" message="" err=<
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:30:57.546509    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main" containerID="cri-o://acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"
Sep 17 19:30:57.546817 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.546560    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main" containerID="cri-o://acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d" gracePeriod=60
Sep 17 19:30:57.711818 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.711777    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:30:57.818996 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.818970    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:57.844456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.844395    2581 generic.go:356] "Generic (PLEG): container finished" podID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerID="acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d" exitCode=0
Sep 17 19:30:57.844622 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.844495    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"
Sep 17 19:30:57.844622 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.844470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerDied","Data":"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"}
Sep 17 19:30:57.844622 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.844616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5" event={"ID":"76b76c37-7916-41ae-9b61-f30a90197cb9","Type":"ContainerDied","Data":"e5d832662eadcbaf7d3d931b9fb104194538e456f7b60d886d3785a57df9d94f"}
Sep 17 19:30:57.844804 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.844652    2581 scope.go:117] "RemoveContainer" containerID="acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"
Sep 17 19:30:57.857472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.857445    2581 scope.go:117] "RemoveContainer" containerID="9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"
Sep 17 19:30:57.866187 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.866160    2581 scope.go:117] "RemoveContainer" containerID="acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"
Sep 17 19:30:57.866670 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:30:57.866641    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d\": container with ID starting with acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d not found: ID does not exist" containerID="acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"
Sep 17 19:30:57.866760 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.866680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d"} err="failed to get container status \"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d\": rpc error: code = NotFound desc = could not find container \"acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d\": container with ID starting with acf9a2004e0428944f698c20ef802c0d132befd43a7a8d989cf68c594807a96d not found: ID does not exist"
Sep 17 19:30:57.866760 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.866698    2581 scope.go:117] "RemoveContainer" containerID="9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"
Sep 17 19:30:57.867029 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:30:57.867000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c\": container with ID starting with 9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c not found: ID does not exist" containerID="9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"
Sep 17 19:30:57.867088 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.867034    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c"} err="failed to get container status \"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c\": rpc error: code = NotFound desc = could not find container \"9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c\": container with ID starting with 9328270daabdc58f1c7eafe952d100072e610a06fa7d343005c0c016551fd75c not found: ID does not exist"
Sep 17 19:30:57.943569 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.943752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.943752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xnf8\" (UniqueName: \"kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.943752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.943752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.943752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.944013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.944013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home\") pod \"76b76c37-7916-41ae-9b61-f30a90197cb9\" (UID: \"76b76c37-7916-41ae-9b61-f30a90197cb9\") "
Sep 17 19:30:57.944013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.944013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.943947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.944557 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.944092    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:57.944557 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.944112    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:57.944557 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.944516    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:57.944948 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.944920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home" (OuterVolumeSpecName: "home") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.946252 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.946223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm" (OuterVolumeSpecName: "dshm") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.946252 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.946236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8" (OuterVolumeSpecName: "kube-api-access-8xnf8") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "kube-api-access-8xnf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:57.946474 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.946312    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:57.964781 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:57.964739    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76b76c37-7916-41ae-9b61-f30a90197cb9" (UID: "76b76c37-7916-41ae-9b61-f30a90197cb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045173    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b76c37-7916-41ae-9b61-f30a90197cb9-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045211    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnf8\" (UniqueName: \"kubernetes.io/projected/76b76c37-7916-41ae-9b61-f30a90197cb9-kube-api-access-8xnf8\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045224    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b76c37-7916-41ae-9b61-f30a90197cb9-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045239    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045251    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.045285 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.045262    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b76c37-7916-41ae-9b61-f30a90197cb9-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.158871 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.158842    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:30:58.162837 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:30:58.162814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-n27f5"]
Sep 17 19:31:00.137674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:00.137634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" path="/var/lib/kubelet/pods/76b76c37-7916-41ae-9b61-f30a90197cb9/volumes"
Sep 17 19:31:02.671393 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:02.671353    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:31:03.062642 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:03.062553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:31:08.065158 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.065088    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:31:08.126732 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.126696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs\") pod \"ff5954b1-528e-4e6c-b9f9-b7964183d394\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") "
Sep 17 19:31:08.126911 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.126760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-725pr\" (UniqueName: \"kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr\") pod \"ff5954b1-528e-4e6c-b9f9-b7964183d394\" (UID: \"ff5954b1-528e-4e6c-b9f9-b7964183d394\") "
Sep 17 19:31:08.129193 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.129162    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr" (OuterVolumeSpecName: "kube-api-access-725pr") pod "ff5954b1-528e-4e6c-b9f9-b7964183d394" (UID: "ff5954b1-528e-4e6c-b9f9-b7964183d394"). InnerVolumeSpecName "kube-api-access-725pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:31:08.129312 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.129210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff5954b1-528e-4e6c-b9f9-b7964183d394" (UID: "ff5954b1-528e-4e6c-b9f9-b7964183d394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:31:08.227454 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.227400    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff5954b1-528e-4e6c-b9f9-b7964183d394-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:31:08.227454 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.227452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725pr\" (UniqueName: \"kubernetes.io/projected/ff5954b1-528e-4e6c-b9f9-b7964183d394-kube-api-access-725pr\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:31:08.895544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.895506    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerID="4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc" exitCode=0
Sep 17 19:31:08.895718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.895560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" event={"ID":"ff5954b1-528e-4e6c-b9f9-b7964183d394","Type":"ContainerDied","Data":"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"}
Sep 17 19:31:08.895718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.895604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z" event={"ID":"ff5954b1-528e-4e6c-b9f9-b7964183d394","Type":"ContainerDied","Data":"af865732048104e33c71892526448bad4e380fdfc93f4adf9ffe70753a16d08b"}
Sep 17 19:31:08.895718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.895626    2581 scope.go:117] "RemoveContainer" containerID="4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"
Sep 17 19:31:08.895718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.895563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"
Sep 17 19:31:08.904459 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.904439    2581 scope.go:117] "RemoveContainer" containerID="4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"
Sep 17 19:31:08.904755 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:31:08.904726    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc\": container with ID starting with 4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc not found: ID does not exist" containerID="4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"
Sep 17 19:31:08.904823 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.904763    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc"} err="failed to get container status \"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc\": rpc error: code = NotFound desc = could not find container \"4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc\": container with ID starting with 4e48063f0ff913361753027e22b98c50b11215b4fc7f32effb6151c382ab4bcc not found: ID does not exist"
Sep 17 19:31:08.913891 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.913861    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:31:08.915213 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:08.915192    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlh5z"]
Sep 17 19:31:10.138143 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:10.138115    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" path="/var/lib/kubelet/pods/ff5954b1-528e-4e6c-b9f9-b7964183d394/volumes"
Sep 17 19:31:10.181918 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:10.181886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:31:10.185095 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:10.185074    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:31:12.562887 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:12.562823    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" containerID="cri-o://6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3" gracePeriod=45
Sep 17 19:31:12.671460 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:12.671402    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:31:22.671253 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:22.671172    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:31:22.671665 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:22.671288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:31:27.184761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.184726    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185046    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185070    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="storage-initializer"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185076    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="storage-initializer"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185091    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185138    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b76c37-7916-41ae-9b61-f30a90197cb9" containerName="main"
Sep 17 19:31:27.185229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.185147    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5954b1-528e-4e6c-b9f9-b7964183d394" containerName="main"
Sep 17 19:31:27.189374 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.189356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.193224 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.193199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 19:31:27.193402 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.193197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-2bbfj\""
Sep 17 19:31:27.193402 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.193267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 19:31:27.193402 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.193199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 19:31:27.193402 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.193210    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-fd2sj\""
Sep 17 19:31:27.220858 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.220828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:31:27.282164 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.282127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.282350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.282200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695gg\" (UniqueName: \"kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.383489 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.383449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.383662 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.383508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695gg\" (UniqueName: \"kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.386203 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.386183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.391690 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.391668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-695gg\" (UniqueName: \"kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.500033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.499943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:27.779971 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.779943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:31:27.782525 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:31:27.782486    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01822176_e3cf_4bd6_94d9_85af3c07a705.slice/crio-9dc66c7c8ec7f4c8f038795fc8a6fc84bc65d66b500bdde107e3f9ad817cc6eb WatchSource:0}: Error finding container 9dc66c7c8ec7f4c8f038795fc8a6fc84bc65d66b500bdde107e3f9ad817cc6eb: Status 404 returned error can't find the container with id 9dc66c7c8ec7f4c8f038795fc8a6fc84bc65d66b500bdde107e3f9ad817cc6eb
Sep 17 19:31:27.989869 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.989823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" event={"ID":"01822176-e3cf-4bd6-94d9-85af3c07a705","Type":"ContainerStarted","Data":"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"}
Sep 17 19:31:27.990081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.989926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" event={"ID":"01822176-e3cf-4bd6-94d9-85af3c07a705","Type":"ContainerStarted","Data":"9dc66c7c8ec7f4c8f038795fc8a6fc84bc65d66b500bdde107e3f9ad817cc6eb"}
Sep 17 19:31:27.990081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:27.989990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:31:28.013216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:28.013160    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podStartSLOduration=1.013144259 podStartE2EDuration="1.013144259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:28.011570366 +0000 UTC m=+1520.025140527" watchObservedRunningTime="2026-09-17 19:31:28.013144259 +0000 UTC m=+1520.026714420"
Sep 17 19:31:32.671593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:32.671557    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:31:42.670187 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:31:42.670154    2581 logging.go:55] [core] [Channel #530 SubChannel #531]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.47:9003", ServerName: "10.132.0.47:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.47:9003: connect: connection refused"
Sep 17 19:31:42.815690 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:42.815662    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:31:42.924999 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:42.924963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs\") pod \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") "
Sep 17 19:31:42.925189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:42.925041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nnz7\" (UniqueName: \"kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7\") pod \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\" (UID: \"193c1b68-e3c5-4d0c-b814-efd93f86abfb\") "
Sep 17 19:31:42.927332 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:42.927295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "193c1b68-e3c5-4d0c-b814-efd93f86abfb" (UID: "193c1b68-e3c5-4d0c-b814-efd93f86abfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:31:42.927462 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:42.927302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7" (OuterVolumeSpecName: "kube-api-access-6nnz7") pod "193c1b68-e3c5-4d0c-b814-efd93f86abfb" (UID: "193c1b68-e3c5-4d0c-b814-efd93f86abfb"). InnerVolumeSpecName "kube-api-access-6nnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:31:43.025643 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.025611    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193c1b68-e3c5-4d0c-b814-efd93f86abfb-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:31:43.025643 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.025639    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnz7\" (UniqueName: \"kubernetes.io/projected/193c1b68-e3c5-4d0c-b814-efd93f86abfb-kube-api-access-6nnz7\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:31:43.060476 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.060445    2581 generic.go:356] "Generic (PLEG): container finished" podID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerID="6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3" exitCode=0
Sep 17 19:31:43.060625 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.060526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" event={"ID":"193c1b68-e3c5-4d0c-b814-efd93f86abfb","Type":"ContainerDied","Data":"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"}
Sep 17 19:31:43.060625 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.060533    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"
Sep 17 19:31:43.060625 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.060563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" event={"ID":"193c1b68-e3c5-4d0c-b814-efd93f86abfb","Type":"ContainerDied","Data":"c9237c9a95227f8b1aa2ad5641da0de51395c5a45d0f617546a1cc12c0a396fb"}
Sep 17 19:31:43.060625 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.060586    2581 scope.go:117] "RemoveContainer" containerID="6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"
Sep 17 19:31:43.069632 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.069613    2581 scope.go:117] "RemoveContainer" containerID="6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"
Sep 17 19:31:43.069934 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:31:43.069914    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3\": container with ID starting with 6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3 not found: ID does not exist" containerID="6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"
Sep 17 19:31:43.069980 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.069945    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3"} err="failed to get container status \"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3\": rpc error: code = NotFound desc = could not find container \"6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3\": container with ID starting with 6532dae8d777c5664bc0303bcaf2d1b5841c2fa149d68614531c6c791ce0fbe3 not found: ID does not exist"
Sep 17 19:31:43.081550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.081525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:31:43.087321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.087299    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws"]
Sep 17 19:31:43.670034 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:43.669995    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dmr5ws" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.47:9003\" within 1s: context deadline exceeded"
Sep 17 19:31:44.137173 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:44.137093    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" path="/var/lib/kubelet/pods/193c1b68-e3c5-4d0c-b814-efd93f86abfb/volumes"
Sep 17 19:31:58.997587 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:31:58.997551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:33:43.702113 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:33:43.702075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:33:48.997723 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:33:48.997673    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:58.725086 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:33:58.725047    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main" containerID="cri-o://257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469" gracePeriod=45
Sep 17 19:33:58.997461 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:33:58.997362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:08.996836 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:08.996801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:08.997395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:08.996932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:34:18.996656 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:18.996615    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:28.975333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:28.975309    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:34:29.102372 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.102281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs\") pod \"01822176-e3cf-4bd6-94d9-85af3c07a705\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") "
Sep 17 19:34:29.102372 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.102315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-695gg\" (UniqueName: \"kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg\") pod \"01822176-e3cf-4bd6-94d9-85af3c07a705\" (UID: \"01822176-e3cf-4bd6-94d9-85af3c07a705\") "
Sep 17 19:34:29.104732 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.104701    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01822176-e3cf-4bd6-94d9-85af3c07a705" (UID: "01822176-e3cf-4bd6-94d9-85af3c07a705"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:29.104732 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.104719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg" (OuterVolumeSpecName: "kube-api-access-695gg") pod "01822176-e3cf-4bd6-94d9-85af3c07a705" (UID: "01822176-e3cf-4bd6-94d9-85af3c07a705"). InnerVolumeSpecName "kube-api-access-695gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:29.203013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.202976    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01822176-e3cf-4bd6-94d9-85af3c07a705-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:34:29.203013 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.203008    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-695gg\" (UniqueName: \"kubernetes.io/projected/01822176-e3cf-4bd6-94d9-85af3c07a705-kube-api-access-695gg\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:34:29.809015 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.808978    2581 generic.go:356] "Generic (PLEG): container finished" podID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerID="257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469" exitCode=0
Sep 17 19:34:29.809185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.809024    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"
Sep 17 19:34:29.809185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.809055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" event={"ID":"01822176-e3cf-4bd6-94d9-85af3c07a705","Type":"ContainerDied","Data":"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"}
Sep 17 19:34:29.809185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.809107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg" event={"ID":"01822176-e3cf-4bd6-94d9-85af3c07a705","Type":"ContainerDied","Data":"9dc66c7c8ec7f4c8f038795fc8a6fc84bc65d66b500bdde107e3f9ad817cc6eb"}
Sep 17 19:34:29.809185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.809132    2581 scope.go:117] "RemoveContainer" containerID="257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"
Sep 17 19:34:29.818048 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.818031    2581 scope.go:117] "RemoveContainer" containerID="257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"
Sep 17 19:34:29.818323 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:34:29.818305    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469\": container with ID starting with 257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469 not found: ID does not exist" containerID="257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"
Sep 17 19:34:29.818367 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.818331    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469"} err="failed to get container status \"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469\": rpc error: code = NotFound desc = could not find container \"257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469\": container with ID starting with 257fbfc99ad64e8a25cbe2adec170d74603e9cdcdc6623ab76f0c342ce193469 not found: ID does not exist"
Sep 17 19:34:29.834790 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.832193    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:34:29.839506 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:29.839477    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-x47tg"]
Sep 17 19:34:30.138888 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:30.138810    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" path="/var/lib/kubelet/pods/01822176-e3cf-4bd6-94d9-85af3c07a705/volumes"
Sep 17 19:34:34.026493 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.026460    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:34:34.026960 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.026941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main"
Sep 17 19:34:34.027005 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.026966    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main"
Sep 17 19:34:34.027005 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.026977    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main"
Sep 17 19:34:34.027005 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.026985    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main"
Sep 17 19:34:34.027098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.027071    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="193c1b68-e3c5-4d0c-b814-efd93f86abfb" containerName="main"
Sep 17 19:34:34.027098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.027085    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="01822176-e3cf-4bd6-94d9-85af3c07a705" containerName="main"
Sep 17 19:34:34.031410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.031389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.034194 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.034173    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 19:34:34.034337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.034211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:34:34.034337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.034216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-9wbm2\""
Sep 17 19:34:34.034337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.034263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 19:34:34.047065 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.047035    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:34:34.148094 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148094 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlfh\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.148504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.148408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249444 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlfh\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249638 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.249899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.249876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.250083 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.250045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.250142 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.250108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.250180 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.250147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.250231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.250184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.250882 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.250858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.252117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.252083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.252324 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.252309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.260504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.260404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.260608 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.260531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlfh\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh\") pod \"router-gateway-1-openshift-default-545f9b8b4c-kbpgg\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.342714 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.342623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:34.626913 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.626828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:34:34.630355 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:34:34.630327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b06244_09bd_47d1_ae95_c5ab35a3b526.slice/crio-c773b87d988fe06cb43dfd0b37b2060cf9e774310912a70cd80c76123c262d92 WatchSource:0}: Error finding container c773b87d988fe06cb43dfd0b37b2060cf9e774310912a70cd80c76123c262d92: Status 404 returned error can't find the container with id c773b87d988fe06cb43dfd0b37b2060cf9e774310912a70cd80c76123c262d92
Sep 17 19:34:34.632256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.632240    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:34:34.835145 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:34.835108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerStarted","Data":"c773b87d988fe06cb43dfd0b37b2060cf9e774310912a70cd80c76123c262d92"}
Sep 17 19:34:37.552592 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:37.552548    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:37.552867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:37.552629    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:37.552867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:37.552659    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:37.851399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:37.851316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerStarted","Data":"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"}
Sep 17 19:34:37.871906 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:37.871854    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podStartSLOduration=1.951898876 podStartE2EDuration="4.871836967s" totalImagesPullingTime="2.919938091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:37.870598222 +0000 UTC m=+1709.884168419" watchObservedRunningTime="2026-09-17 19:34:37.871836967 +0000 UTC m=+1709.885407127"
Sep 17 19:34:38.343452 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:38.343402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:38.348080 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:38.348057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:38.856091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:38.856057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:38.857204 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:38.857187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:45.448833 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.448794    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:34:45.452847 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.452824    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.456058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.456029    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 19:34:45.456058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.456049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-s2p9s\""
Sep 17 19:34:45.461911 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.461890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:34:45.552958 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.552926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.553138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.552966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblxj\" (UniqueName: \"kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.553138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.553011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.553138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.553029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.553138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.553062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.553138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.553085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.653821 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.653784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.653833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.653870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.653909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pblxj\" (UniqueName: \"kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.653975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.654000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.654341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654461 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.654372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.654461 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.654404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.656394 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.656365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.656596 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.656580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.664766 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.664740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblxj\" (UniqueName: \"kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:45.764911 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:45.764818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:46.037079 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:46.037055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:34:46.039340 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:34:46.039312    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86ce88b_54b4_4f31_a3d7_389875b969ff.slice/crio-d614ef3b46dd09a19d62dea371693e8e8ebf863723217e1beba41f1de62631ef WatchSource:0}: Error finding container d614ef3b46dd09a19d62dea371693e8e8ebf863723217e1beba41f1de62631ef: Status 404 returned error can't find the container with id d614ef3b46dd09a19d62dea371693e8e8ebf863723217e1beba41f1de62631ef
Sep 17 19:34:46.893095 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:46.893054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" event={"ID":"a86ce88b-54b4-4f31-a3d7-389875b969ff","Type":"ContainerStarted","Data":"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"}
Sep 17 19:34:46.893500 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:46.893129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" event={"ID":"a86ce88b-54b4-4f31-a3d7-389875b969ff","Type":"ContainerStarted","Data":"d614ef3b46dd09a19d62dea371693e8e8ebf863723217e1beba41f1de62631ef"}
Sep 17 19:34:46.915313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:46.915264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" podStartSLOduration=1.91524844 podStartE2EDuration="1.91524844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:46.914196022 +0000 UTC m=+1718.927766186" watchObservedRunningTime="2026-09-17 19:34:46.91524844 +0000 UTC m=+1718.928818600"
Sep 17 19:34:47.766019 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:47.765931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:54.897603 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:54.897558    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:34:55.765471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:55.765436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:55.777899 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:55.777866    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:55.948771 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:55.948738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:34:56.938407 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.938376    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerID="8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1" exitCode=0
Sep 17 19:34:56.938631 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.938449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerDied","Data":"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"}
Sep 17 19:34:56.938962 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.938944    2581 scope.go:117] "RemoveContainer" containerID="8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"
Sep 17 19:34:56.939870 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.939847    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:56.939925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.939914    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:56.939960 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:56.939942    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:34:57.948101 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:57.948048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerStarted","Data":"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"}
Sep 17 19:34:58.344020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:58.343921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:34:58.344440 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:58.344323    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 19:34:59.343360 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:34:59.343305    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 19:35:00.343613 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:00.343566    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 19:35:01.343981 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:01.343923    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 19:35:02.343545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:02.343499    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 19:35:03.346810 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:03.346779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:35:03.347204 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:03.347134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:35:03.347940 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:03.347918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:35:57.484320 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.484231    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:35:57.493907 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:35:57.493864    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:35:57.493907 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:57.493907 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:57.493907 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555" cmd=["/bin/sleep","15"]
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.493923    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" message="" err=<
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:35:57.493937    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerName="main" containerID="cri-o://67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"
Sep 17 19:35:57.494190 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.493985    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerName="main" containerID="cri-o://67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555" gracePeriod=60
Sep 17 19:35:57.875741 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.875710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:35:57.992240 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pblxj\" (UniqueName: \"kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992484 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home\") pod \"a86ce88b-54b4-4f31-a3d7-389875b969ff\" (UID: \"a86ce88b-54b4-4f31-a3d7-389875b969ff\") "
Sep 17 19:35:57.992765 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:57.992765 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:57.992765 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.992710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home" (OuterVolumeSpecName: "home") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:57.994801 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.994774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm" (OuterVolumeSpecName: "dshm") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:57.994906 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.994799    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:35:57.994906 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:57.994811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj" (OuterVolumeSpecName: "kube-api-access-pblxj") pod "a86ce88b-54b4-4f31-a3d7-389875b969ff" (UID: "a86ce88b-54b4-4f31-a3d7-389875b969ff"). InnerVolumeSpecName "kube-api-access-pblxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093193    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093226    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093236    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093244    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86ce88b-54b4-4f31-a3d7-389875b969ff-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093252    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pblxj\" (UniqueName: \"kubernetes.io/projected/a86ce88b-54b4-4f31-a3d7-389875b969ff-kube-api-access-pblxj\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.093281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.093260    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86ce88b-54b4-4f31-a3d7-389875b969ff-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:35:58.223806 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.223770    2581 generic.go:356] "Generic (PLEG): container finished" podID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerID="67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555" exitCode=0
Sep 17 19:35:58.224006 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.223817    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"
Sep 17 19:35:58.224006 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.223862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" event={"ID":"a86ce88b-54b4-4f31-a3d7-389875b969ff","Type":"ContainerDied","Data":"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"}
Sep 17 19:35:58.224006 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.223919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855" event={"ID":"a86ce88b-54b4-4f31-a3d7-389875b969ff","Type":"ContainerDied","Data":"d614ef3b46dd09a19d62dea371693e8e8ebf863723217e1beba41f1de62631ef"}
Sep 17 19:35:58.224006 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.223953    2581 scope.go:117] "RemoveContainer" containerID="67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"
Sep 17 19:35:58.235043 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.234950    2581 scope.go:117] "RemoveContainer" containerID="67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"
Sep 17 19:35:58.235404 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:35:58.235332    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555\": container with ID starting with 67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555 not found: ID does not exist" containerID="67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"
Sep 17 19:35:58.235404 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.235366    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555"} err="failed to get container status \"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555\": rpc error: code = NotFound desc = could not find container \"67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555\": container with ID starting with 67b07ccfb612c633b8d1e10066268bd8ca421fee035e6207b38a3d17b6bb8555 not found: ID does not exist"
Sep 17 19:35:58.239440 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.239404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:35:58.243292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:35:58.243270    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589975x855"]
Sep 17 19:36:00.137830 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:00.137800    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" path="/var/lib/kubelet/pods/a86ce88b-54b4-4f31-a3d7-389875b969ff/volumes"
Sep 17 19:36:07.435796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:07.435764    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:36:07.436321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:07.435993    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy" containerID="cri-o://76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838" gracePeriod=30
Sep 17 19:36:10.208712 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:10.208681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:36:10.212903 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:10.212878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:36:12.576337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.576298    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:36:12.576728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.576640    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerName="main"
Sep 17 19:36:12.576728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.576652    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerName="main"
Sep 17 19:36:12.576728 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.576724    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86ce88b-54b4-4f31-a3d7-389875b969ff" containerName="main"
Sep 17 19:36:12.580121 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.580094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.583316 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 19:36:12.583549 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:36:12.583644 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583397    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-fs8cc\""
Sep 17 19:36:12.583644 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583402    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 19:36:12.583760 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 19:36:12.583949 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.583928    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 19:36:12.597303 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.597278    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:36:12.704009 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.703939    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:36:12.721410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9mb\" (UniqueName: \"kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721763 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.721763 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.721619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.822348 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822485    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjlfh\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.822565 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket\") pod \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\" (UID: \"e3b06244-09bd-47d1-ae95-c5ab35a3b526\") "
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822672    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:36:12.823020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.822789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9mb\" (UniqueName: \"kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823329    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istiod-ca-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823345    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.823358 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823359    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-data\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.823925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:12.823925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.823925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:12.824153 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.823951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.824213 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.824175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.825791 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.825754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:12.826582 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.826543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh" (OuterVolumeSpecName: "kube-api-access-cjlfh") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "kube-api-access-cjlfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:36:12.826704 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.826624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:36:12.826776 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.826762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.826900 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.826876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3b06244-09bd-47d1-ae95-c5ab35a3b526" (UID: "e3b06244-09bd-47d1-ae95-c5ab35a3b526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:36:12.827606 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.827566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.834648 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.834610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9mb\" (UniqueName: \"kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.836168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.836143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:36:12.836553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.836538    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:36:12.836630 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.836555    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:36:12.836630 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.836624    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:36:12.836736 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.836633    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:36:12.840847 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.840786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:12.843036 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.843013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-qk827\""
Sep 17 19:36:12.850336 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.850309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:36:12.893061 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.893029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:12.924244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:12.924392 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq45\" (UniqueName: \"kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:12.924472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924437    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-workload-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.924472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924452    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-podinfo\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.924472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924460    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-token\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.924472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924467    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlfh\" (UniqueName: \"kubernetes.io/projected/e3b06244-09bd-47d1-ae95-c5ab35a3b526-kube-api-access-cjlfh\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.924595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924475    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-istio-envoy\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:12.924595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:12.924483    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b06244-09bd-47d1-ae95-c5ab35a3b526-credential-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:36:13.025658 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.025566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:13.025658 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.025646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq45\" (UniqueName: \"kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:13.028309 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.028285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:13.033682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.033660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq45\" (UniqueName: \"kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:13.154557 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.154525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:13.170721 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.170696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:36:13.172793 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:36:13.172762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bfb45_6d14_4ea9_8487_527f2e8b366c.slice/crio-132ccd6ebf94885ecc61cf22d97169e267788f6d0a6da7f023f5bfb1c26d264e WatchSource:0}: Error finding container 132ccd6ebf94885ecc61cf22d97169e267788f6d0a6da7f023f5bfb1c26d264e: Status 404 returned error can't find the container with id 132ccd6ebf94885ecc61cf22d97169e267788f6d0a6da7f023f5bfb1c26d264e
Sep 17 19:36:13.293548 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.293462    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerID="76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838" exitCode=0
Sep 17 19:36:13.293548 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.293513    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"
Sep 17 19:36:13.293752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.293545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerDied","Data":"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"}
Sep 17 19:36:13.293752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.293598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg" event={"ID":"e3b06244-09bd-47d1-ae95-c5ab35a3b526","Type":"ContainerDied","Data":"c773b87d988fe06cb43dfd0b37b2060cf9e774310912a70cd80c76123c262d92"}
Sep 17 19:36:13.293752 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.293623    2581 scope.go:117] "RemoveContainer" containerID="76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"
Sep 17 19:36:13.295468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.295439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerStarted","Data":"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"}
Sep 17 19:36:13.295600 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.295495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerStarted","Data":"132ccd6ebf94885ecc61cf22d97169e267788f6d0a6da7f023f5bfb1c26d264e"}
Sep 17 19:36:13.303375 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.303348    2581 scope.go:117] "RemoveContainer" containerID="8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"
Sep 17 19:36:13.312101 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.312082    2581 scope.go:117] "RemoveContainer" containerID="76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"
Sep 17 19:36:13.312433 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:36:13.312394    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838\": container with ID starting with 76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838 not found: ID does not exist" containerID="76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"
Sep 17 19:36:13.312725 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.312704    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838"} err="failed to get container status \"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838\": rpc error: code = NotFound desc = could not find container \"76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838\": container with ID starting with 76fe64e4e6abe74e66ec2a792cbea937621cc83dec11fe5b8a2a543bd40a9838 not found: ID does not exist"
Sep 17 19:36:13.312768 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.312729    2581 scope.go:117] "RemoveContainer" containerID="8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"
Sep 17 19:36:13.313010 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:36:13.312988    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1\": container with ID starting with 8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1 not found: ID does not exist" containerID="8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"
Sep 17 19:36:13.313076 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.313015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1"} err="failed to get container status \"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1\": rpc error: code = NotFound desc = could not find container \"8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1\": container with ID starting with 8c6f2eea89b578bb98d61428a1845741205ded919adfe4b5839f12632761a1c1 not found: ID does not exist"
Sep 17 19:36:13.334068 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.334039    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:36:13.342127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.342095    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-545f9b8b4c-kbpgg"]
Sep 17 19:36:13.433470 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:13.433407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:36:13.437753 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:36:13.437729    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2c622d_34de_4d90_9d0a_d26a96c198e7.slice/crio-c16751023e71bb880e53559a0d1036bd3526f40a532b1d62cf94814af1e06ed8 WatchSource:0}: Error finding container c16751023e71bb880e53559a0d1036bd3526f40a532b1d62cf94814af1e06ed8: Status 404 returned error can't find the container with id c16751023e71bb880e53559a0d1036bd3526f40a532b1d62cf94814af1e06ed8
Sep 17 19:36:14.138805 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:14.138773    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" path="/var/lib/kubelet/pods/e3b06244-09bd-47d1-ae95-c5ab35a3b526/volumes"
Sep 17 19:36:14.302581 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:14.302548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" event={"ID":"5b2c622d-34de-4d90-9d0a-d26a96c198e7","Type":"ContainerStarted","Data":"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"}
Sep 17 19:36:14.302747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:14.302599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" event={"ID":"5b2c622d-34de-4d90-9d0a-d26a96c198e7","Type":"ContainerStarted","Data":"c16751023e71bb880e53559a0d1036bd3526f40a532b1d62cf94814af1e06ed8"}
Sep 17 19:36:14.302747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:14.302625    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:14.320306 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:14.320246    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podStartSLOduration=2.320223102 podStartE2EDuration="2.320223102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:14.318817352 +0000 UTC m=+1806.332387513" watchObservedRunningTime="2026-09-17 19:36:14.320223102 +0000 UTC m=+1806.333793285"
Sep 17 19:36:16.312664 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:16.312629    2581 generic.go:356] "Generic (PLEG): container finished" podID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerID="d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674" exitCode=0
Sep 17 19:36:16.313039 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:16.312676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerDied","Data":"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"}
Sep 17 19:36:45.309309 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:45.309277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:36:45.482301 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:45.482266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerStarted","Data":"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"}
Sep 17 19:36:45.504646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:45.504589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podStartSLOduration=2.421203925 podStartE2EDuration="33.504572638s" totalImagesPullingTime="28.762150087s" totalInitContainerRuntime="2.321218626s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:45.502213161 +0000 UTC m=+1837.515783327" watchObservedRunningTime="2026-09-17 19:36:45.504572638 +0000 UTC m=+1837.518142799"
Sep 17 19:36:45.894028 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:45.893944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:52.893591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:52.893542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:36:52.895275 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:36:52.895248    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:02.894346 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:02.894301    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:12.894151 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:12.894109    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:22.893873 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:22.893778    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:32.893619 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:32.893576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:42.894294 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:42.894235    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:37:52.894264 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:37:52.894223    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:38:02.893892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:02.893850    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:38:12.893652 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:12.893606    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:38:22.894493 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:22.894409    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:38:32.894518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:32.894468    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:38:42.904288 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:42.904254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:38:42.912027 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:42.911995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:38:48.202366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:48.202326    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:38:48.210270 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:48.210242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:38:55.309269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:38:55.309180    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:03.227463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.227406    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main" containerID="cri-o://3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140" gracePeriod=45
Sep 17 19:39:03.231091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.231018    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main" containerID="cri-o://23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801" gracePeriod=45
Sep 17 19:39:03.889307 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.889283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:39:03.928071 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928071 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k9mb\" (UniqueName: \"kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928325 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928454 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm\") pod \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\" (UID: \"e21bfb45-6d14-4ea9-8487-527f2e8b366c\") "
Sep 17 19:39:03.928830 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928718    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:03.928887 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.928848    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:03.930027 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.929089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:03.930027 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.929122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home" (OuterVolumeSpecName: "home") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:03.930847 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.930824    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:03.931585 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.931562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm" (OuterVolumeSpecName: "dshm") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:03.931585 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.931575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb" (OuterVolumeSpecName: "kube-api-access-2k9mb") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "kube-api-access-2k9mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:03.947901 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.947866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:03.955747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:03.955722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e21bfb45-6d14-4ea9-8487-527f2e8b366c" (UID: "e21bfb45-6d14-4ea9-8487-527f2e8b366c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029313    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029344    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21bfb45-6d14-4ea9-8487-527f2e8b366c-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029354    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029361    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21bfb45-6d14-4ea9-8487-527f2e8b366c-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029369    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9mb\" (UniqueName: \"kubernetes.io/projected/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kube-api-access-2k9mb\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029382    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.029406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.029394    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e21bfb45-6d14-4ea9-8487-527f2e8b366c-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:04.143294 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.143258    2581 generic.go:356] "Generic (PLEG): container finished" podID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerID="3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140" exitCode=0
Sep 17 19:39:04.143481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.143293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerDied","Data":"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"}
Sep 17 19:39:04.143481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.143314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"
Sep 17 19:39:04.143481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.143348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb" event={"ID":"e21bfb45-6d14-4ea9-8487-527f2e8b366c","Type":"ContainerDied","Data":"132ccd6ebf94885ecc61cf22d97169e267788f6d0a6da7f023f5bfb1c26d264e"}
Sep 17 19:39:04.143481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.143371    2581 scope.go:117] "RemoveContainer" containerID="3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"
Sep 17 19:39:04.153284 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.153266    2581 scope.go:117] "RemoveContainer" containerID="d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"
Sep 17 19:39:04.161371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.161343    2581 scope.go:117] "RemoveContainer" containerID="3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"
Sep 17 19:39:04.161750 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:39:04.161728    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140\": container with ID starting with 3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140 not found: ID does not exist" containerID="3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"
Sep 17 19:39:04.161859 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.161762    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140"} err="failed to get container status \"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140\": rpc error: code = NotFound desc = could not find container \"3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140\": container with ID starting with 3de0773f18c10b1c3db355405b44db5f8aad0ca2796263f1d186940912883140 not found: ID does not exist"
Sep 17 19:39:04.161859 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.161789    2581 scope.go:117] "RemoveContainer" containerID="d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"
Sep 17 19:39:04.162031 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.162005    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:39:04.162210 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:39:04.162193    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674\": container with ID starting with d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674 not found: ID does not exist" containerID="d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"
Sep 17 19:39:04.162260 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.162215    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674"} err="failed to get container status \"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674\": rpc error: code = NotFound desc = could not find container \"d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674\": container with ID starting with d441daa4d7a84e30266039a899856d48b66d502db4b055f1b0332f0debdcd674 not found: ID does not exist"
Sep 17 19:39:04.166251 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:04.166230    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7ccb98d9f6zfswb"]
Sep 17 19:39:05.309410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:05.309371    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:06.137140 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:06.137103    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" path="/var/lib/kubelet/pods/e21bfb45-6d14-4ea9-8487-527f2e8b366c/volumes"
Sep 17 19:39:15.309252 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:15.309219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:15.309672 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:15.309337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:39:17.978721 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.978687    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:39:17.979175 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979164    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:39:17.979244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979182    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b06244-09bd-47d1-ae95-c5ab35a3b526" containerName="istio-proxy"
Sep 17 19:39:17.979244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979196    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main"
Sep 17 19:39:17.979244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979205    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main"
Sep 17 19:39:17.979406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979242    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="storage-initializer"
Sep 17 19:39:17.979406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979252    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="storage-initializer"
Sep 17 19:39:17.979406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.979333    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21bfb45-6d14-4ea9-8487-527f2e8b366c" containerName="main"
Sep 17 19:39:17.984669 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.984643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:17.987040 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.987012    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-jvdqf\""
Sep 17 19:39:17.987716 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.987694    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:39:17.987853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.987770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:39:17.987925 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.987902    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:39:17.987988 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.987959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-fmbwz\""
Sep 17 19:39:17.995350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:17.995328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:39:18.039373 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.039341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.039535 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.039439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8fk\" (UniqueName: \"kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.139827 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.139792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8fk\" (UniqueName: \"kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.139827 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.139830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.142690 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.142665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.148586 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.148560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8fk\" (UniqueName: \"kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.296905 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.296819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:18.572304 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:18.572244    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:39:18.574643 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:39:18.574615    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01801d1_10b6_49c7_8a83_8f11a8b6e311.slice/crio-4a030a48262d19273aaca30ac595f6cdbdcbf78478fbbcc6381bf8d84091201c WatchSource:0}: Error finding container 4a030a48262d19273aaca30ac595f6cdbdcbf78478fbbcc6381bf8d84091201c: Status 404 returned error can't find the container with id 4a030a48262d19273aaca30ac595f6cdbdcbf78478fbbcc6381bf8d84091201c
Sep 17 19:39:19.211764 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:19.211651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" event={"ID":"b01801d1-10b6-49c7-8a83-8f11a8b6e311","Type":"ContainerStarted","Data":"4bc8b023b279ff3c60956927807cc699618767c896de1719975535d5abc114b1"}
Sep 17 19:39:19.211764 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:19.211723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" event={"ID":"b01801d1-10b6-49c7-8a83-8f11a8b6e311","Type":"ContainerStarted","Data":"4a030a48262d19273aaca30ac595f6cdbdcbf78478fbbcc6381bf8d84091201c"}
Sep 17 19:39:19.212321 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:19.211871    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:19.228818 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:19.228769    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podStartSLOduration=2.228748799 podStartE2EDuration="2.228748799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:39:19.227028738 +0000 UTC m=+1991.240598899" watchObservedRunningTime="2026-09-17 19:39:19.228748799 +0000 UTC m=+1991.242318957"
Sep 17 19:39:25.309355 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:25.309314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:33.492998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.492937    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:39:33.570323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.570289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nq45\" (UniqueName: \"kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45\") pod \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") "
Sep 17 19:39:33.570538 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.570345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs\") pod \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\" (UID: \"5b2c622d-34de-4d90-9d0a-d26a96c198e7\") "
Sep 17 19:39:33.572659 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.572629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b2c622d-34de-4d90-9d0a-d26a96c198e7" (UID: "5b2c622d-34de-4d90-9d0a-d26a96c198e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:33.572768 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.572710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45" (OuterVolumeSpecName: "kube-api-access-7nq45") pod "5b2c622d-34de-4d90-9d0a-d26a96c198e7" (UID: "5b2c622d-34de-4d90-9d0a-d26a96c198e7"). InnerVolumeSpecName "kube-api-access-7nq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:33.671854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.671823    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b2c622d-34de-4d90-9d0a-d26a96c198e7-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:33.671854 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:33.671850    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nq45\" (UniqueName: \"kubernetes.io/projected/5b2c622d-34de-4d90-9d0a-d26a96c198e7-kube-api-access-7nq45\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:39:34.280080 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.280045    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerID="23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801" exitCode=0
Sep 17 19:39:34.280274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.280090    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"
Sep 17 19:39:34.280274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.280124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" event={"ID":"5b2c622d-34de-4d90-9d0a-d26a96c198e7","Type":"ContainerDied","Data":"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"}
Sep 17 19:39:34.280274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.280188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht" event={"ID":"5b2c622d-34de-4d90-9d0a-d26a96c198e7","Type":"ContainerDied","Data":"c16751023e71bb880e53559a0d1036bd3526f40a532b1d62cf94814af1e06ed8"}
Sep 17 19:39:34.280274 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.280217    2581 scope.go:117] "RemoveContainer" containerID="23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"
Sep 17 19:39:34.289067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.289044    2581 scope.go:117] "RemoveContainer" containerID="23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"
Sep 17 19:39:34.289335 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:39:34.289317    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801\": container with ID starting with 23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801 not found: ID does not exist" containerID="23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"
Sep 17 19:39:34.289380 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.289342    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801"} err="failed to get container status \"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801\": rpc error: code = NotFound desc = could not find container \"23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801\": container with ID starting with 23bdd588aa7194bc261d49a7454da756250d9b7342dd9d3dd890d4af25129801 not found: ID does not exist"
Sep 17 19:39:34.296406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.296385    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:39:34.300696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:34.300675    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebgvht"]
Sep 17 19:39:36.138802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:36.138760    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" path="/var/lib/kubelet/pods/5b2c622d-34de-4d90-9d0a-d26a96c198e7/volumes"
Sep 17 19:39:44.447940 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.447903    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:39:44.448574 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.448262    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main"
Sep 17 19:39:44.448574 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.448275    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main"
Sep 17 19:39:44.448574 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.448347    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b2c622d-34de-4d90-9d0a-d26a96c198e7" containerName="main"
Sep 17 19:39:44.452890 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.452872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.455503 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.455474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 19:39:44.455626 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.455503    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 19:39:44.455626 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.455519    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 19:39:44.455626 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.455520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-ln4x2\""
Sep 17 19:39:44.467060 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.467038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:39:44.558455 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qjj\" (UniqueName: \"kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558455 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.558698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.558619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qjj\" (UniqueName: \"kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659343 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659663 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659663 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.659772 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.660052 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.660052 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.659991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.661961 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.661935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.662205 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.662184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.671052 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.671029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qjj\" (UniqueName: \"kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:44.763180 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:44.763081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:39:45.040144 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:45.040114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:39:45.078699 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:45.078678    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:39:45.331169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:45.331085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerStarted","Data":"9341f81c4a6dd99e61d1bdcb140f467aa924cfc876ae0197c70f7051a3b548c6"}
Sep 17 19:39:50.217922 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:50.217891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:39:54.374764 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:54.374721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerStarted","Data":"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"}
Sep 17 19:39:55.380762 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:55.380728    2581 generic.go:356] "Generic (PLEG): container finished" podID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerID="ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5" exitCode=0
Sep 17 19:39:55.381186 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:55.380819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerDied","Data":"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"}
Sep 17 19:39:56.388719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:56.388684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerStarted","Data":"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"}
Sep 17 19:39:56.389136 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:56.388750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerStarted","Data":"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"}
Sep 17 19:39:56.412033 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:56.411981    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podStartSLOduration=2.836941211 podStartE2EDuration="12.411963181s" totalImagesPullingTime="8.389439768s" totalInitContainerRuntime="1.185582202s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:39:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:39:56.41038208 +0000 UTC m=+2028.423952254" watchObservedRunningTime="2026-09-17 19:39:56.411963181 +0000 UTC m=+2028.425533342"
Sep 17 19:39:56.763572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:39:56.763525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:40:04.763923 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:04.763867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:40:04.765502 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:04.765474    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:40:14.764577 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:14.764528    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:40:24.764133 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:24.764032    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:40:34.764588 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:34.764537    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:40:39.591024 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:39.590978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:40:40.218495 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:40.218456    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:40:44.764090 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:44.764034    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:40:50.217930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:50.217889    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:40:54.613208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:54.613136    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" containerID="cri-o://4bc8b023b279ff3c60956927807cc699618767c896de1719975535d5abc114b1" gracePeriod=45
Sep 17 19:40:54.763891 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:40:54.763850    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:00.219075 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:00.219030    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:00.219786 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:00.219207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:41:04.764504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:04.764461    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:10.219118 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:10.219072    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:10.237798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:10.237772    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:41:10.243226 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:10.243199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:41:14.763998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:14.763961    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:20.218345 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:20.218308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:24.764105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.764050    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:24.822073 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.822034    2581 generic.go:356] "Generic (PLEG): container finished" podID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerID="4bc8b023b279ff3c60956927807cc699618767c896de1719975535d5abc114b1" exitCode=0
Sep 17 19:41:24.822224 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.822111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" event={"ID":"b01801d1-10b6-49c7-8a83-8f11a8b6e311","Type":"ContainerDied","Data":"4bc8b023b279ff3c60956927807cc699618767c896de1719975535d5abc114b1"}
Sep 17 19:41:24.875072 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.875000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:41:24.941602 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.941564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8fk\" (UniqueName: \"kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk\") pod \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") "
Sep 17 19:41:24.941796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.941682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs\") pod \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\" (UID: \"b01801d1-10b6-49c7-8a83-8f11a8b6e311\") "
Sep 17 19:41:24.943991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.943961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b01801d1-10b6-49c7-8a83-8f11a8b6e311" (UID: "b01801d1-10b6-49c7-8a83-8f11a8b6e311"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:24.944105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:24.943997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk" (OuterVolumeSpecName: "kube-api-access-gt8fk") pod "b01801d1-10b6-49c7-8a83-8f11a8b6e311" (UID: "b01801d1-10b6-49c7-8a83-8f11a8b6e311"). InnerVolumeSpecName "kube-api-access-gt8fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:25.042822 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.042781    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b01801d1-10b6-49c7-8a83-8f11a8b6e311-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:41:25.042822 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.042818    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8fk\" (UniqueName: \"kubernetes.io/projected/b01801d1-10b6-49c7-8a83-8f11a8b6e311-kube-api-access-gt8fk\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:41:25.827929 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.827895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb" event={"ID":"b01801d1-10b6-49c7-8a83-8f11a8b6e311","Type":"ContainerDied","Data":"4a030a48262d19273aaca30ac595f6cdbdcbf78478fbbcc6381bf8d84091201c"}
Sep 17 19:41:25.828406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.827938    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"
Sep 17 19:41:25.828406 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.827967    2581 scope.go:117] "RemoveContainer" containerID="4bc8b023b279ff3c60956927807cc699618767c896de1719975535d5abc114b1"
Sep 17 19:41:25.849943 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.849911    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:41:25.851744 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:25.851719    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheltjfb"]
Sep 17 19:41:26.137777 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:26.137692    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" path="/var/lib/kubelet/pods/b01801d1-10b6-49c7-8a83-8f11a8b6e311/volumes"
Sep 17 19:41:34.021846 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.021816    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:41:34.022313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.022171    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main"
Sep 17 19:41:34.022313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.022182    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main"
Sep 17 19:41:34.022313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.022242    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01801d1-10b6-49c7-8a83-8f11a8b6e311" containerName="main"
Sep 17 19:41:34.025181 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.025157    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.028719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.028678    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:41:34.028719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.028678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-5zt5s\""
Sep 17 19:41:34.028936 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.028765    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-swr7h\""
Sep 17 19:41:34.028936 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.028882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:41:34.028936 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.028930    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:41:34.033850 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.033533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:41:34.121028 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.120992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.121213 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.121054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6j2\" (UniqueName: \"kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.222345 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.222307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.222553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.222370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6j2\" (UniqueName: \"kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.225103 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.225082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.230188 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.230163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6j2\" (UniqueName: \"kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.339527 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.339429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.615791 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.615679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:41:34.618113 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:41:34.618067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eed9cd1_f862_4158_bcdc_e8d7665e08ef.slice/crio-01ad5d8746704e47c59e266668c8ae722da422b7a1885bff83292c9f436d6538 WatchSource:0}: Error finding container 01ad5d8746704e47c59e266668c8ae722da422b7a1885bff83292c9f436d6538: Status 404 returned error can't find the container with id 01ad5d8746704e47c59e266668c8ae722da422b7a1885bff83292c9f436d6538
Sep 17 19:41:34.763741 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.763691    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:34.877229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.877137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" event={"ID":"2eed9cd1-f862-4158-bcdc-e8d7665e08ef","Type":"ContainerStarted","Data":"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"}
Sep 17 19:41:34.877229 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.877200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" event={"ID":"2eed9cd1-f862-4158-bcdc-e8d7665e08ef","Type":"ContainerStarted","Data":"01ad5d8746704e47c59e266668c8ae722da422b7a1885bff83292c9f436d6538"}
Sep 17 19:41:34.877520 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.877249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:41:34.895324 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:34.895263    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podStartSLOduration=1.895242327 podStartE2EDuration="1.895242327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:34.893978546 +0000 UTC m=+2126.907548707" watchObservedRunningTime="2026-09-17 19:41:34.895242327 +0000 UTC m=+2126.908812489"
Sep 17 19:41:44.763992 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:44.763942    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:41:54.774366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:54.774283    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:41:54.782352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:54.782325    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:41:59.549832 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:59.549797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:41:59.551269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:59.551208    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar" containerID="cri-o://70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5" gracePeriod=60
Sep 17 19:41:59.994987 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:59.994957    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp_908c3d5a-aea8-4ae9-a4e1-3df89abfc01e/modelcar/0.log"
Sep 17 19:41:59.995671 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:59.995646    2581 generic.go:356] "Generic (PLEG): container finished" podID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerID="70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5" exitCode=143
Sep 17 19:41:59.995765 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:41:59.995694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerDied","Data":"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"}
Sep 17 19:42:05.882799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:05.882772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:42:14.579351 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:14.579305    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" containerID="cri-o://0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54" gracePeriod=45
Sep 17 19:42:14.774999 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:14.774947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 19:42:15.833388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.833367    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp_908c3d5a-aea8-4ae9-a4e1-3df89abfc01e/modelcar/0.log"
Sep 17 19:42:15.836291 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.836272    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:42:15.881021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.880952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881011    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qjj\" (UniqueName: \"kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs\") pod \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\" (UID: \"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e\") "
Sep 17 19:42:15.881518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:15.881518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881445    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.881518 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.881474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache" (OuterVolumeSpecName: "model-cache") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:15.882336 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.882308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home" (OuterVolumeSpecName: "home") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:15.884177 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.884157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm" (OuterVolumeSpecName: "dshm") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:15.884308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.884284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:42:15.884372 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.884304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj" (OuterVolumeSpecName: "kube-api-access-l7qjj") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "kube-api-access-l7qjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:42:15.892998 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.892973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" (UID: "908c3d5a-aea8-4ae9-a4e1-3df89abfc01e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:15.981986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.981952    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qjj\" (UniqueName: \"kubernetes.io/projected/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-kube-api-access-l7qjj\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.981986 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.981979    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.982198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.982008    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.982198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.982016    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.982198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.982023    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:15.982198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:15.982031    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:42:16.070217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.070188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp_908c3d5a-aea8-4ae9-a4e1-3df89abfc01e/modelcar/0.log"
Sep 17 19:42:16.070880 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.070849    2581 generic.go:356] "Generic (PLEG): container finished" podID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerID="0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54" exitCode=0
Sep 17 19:42:16.071018 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.070900    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"
Sep 17 19:42:16.071018 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.070901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerDied","Data":"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"}
Sep 17 19:42:16.071122 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.071047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" event={"ID":"908c3d5a-aea8-4ae9-a4e1-3df89abfc01e","Type":"ContainerDied","Data":"9341f81c4a6dd99e61d1bdcb140f467aa924cfc876ae0197c70f7051a3b548c6"}
Sep 17 19:42:16.071122 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.071096    2581 scope.go:117] "RemoveContainer" containerID="70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"
Sep 17 19:42:16.081591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.081565    2581 scope.go:117] "RemoveContainer" containerID="0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"
Sep 17 19:42:16.092545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.092512    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:42:16.093540 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.093516    2581 scope.go:117] "RemoveContainer" containerID="ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"
Sep 17 19:42:16.096116 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.096089    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp"]
Sep 17 19:42:16.101791 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.101774    2581 scope.go:117] "RemoveContainer" containerID="70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"
Sep 17 19:42:16.102089 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:42:16.102071    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5\": container with ID starting with 70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5 not found: ID does not exist" containerID="70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"
Sep 17 19:42:16.102138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.102099    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5"} err="failed to get container status \"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5\": rpc error: code = NotFound desc = could not find container \"70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5\": container with ID starting with 70d0a98f3a7569e1f0b45d5824b714d3dbbac14a59c3bae31d4b3438ffa42fc5 not found: ID does not exist"
Sep 17 19:42:16.102138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.102116    2581 scope.go:117] "RemoveContainer" containerID="0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"
Sep 17 19:42:16.102369 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:42:16.102351    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54\": container with ID starting with 0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54 not found: ID does not exist" containerID="0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"
Sep 17 19:42:16.102410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.102377    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54"} err="failed to get container status \"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54\": rpc error: code = NotFound desc = could not find container \"0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54\": container with ID starting with 0f2195fe8d99d86eb895d827240869ff87d14ce14a03bc0d88ce55d98577ed54 not found: ID does not exist"
Sep 17 19:42:16.102410 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.102396    2581 scope.go:117] "RemoveContainer" containerID="ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"
Sep 17 19:42:16.102676 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:42:16.102661    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5\": container with ID starting with ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5 not found: ID does not exist" containerID="ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"
Sep 17 19:42:16.102715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.102681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5"} err="failed to get container status \"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5\": rpc error: code = NotFound desc = could not find container \"ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5\": container with ID starting with ce1f331c4b9bb43f245d3be5e57b56d1b22391d68f53447e100ef0ab6287e4c5 not found: ID does not exist"
Sep 17 19:42:16.138787 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.138700    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" path="/var/lib/kubelet/pods/908c3d5a-aea8-4ae9-a4e1-3df89abfc01e/volumes"
Sep 17 19:42:16.775965 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:16.775920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dlhwgp" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:42:55.813696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.813658    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.813963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar-init"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.813974    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar-init"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.813997    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.814003    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.814012    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.814018    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.814066    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="main"
Sep 17 19:42:55.814277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.814074    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="908c3d5a-aea8-4ae9-a4e1-3df89abfc01e" containerName="modelcar"
Sep 17 19:42:55.817010 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.816989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:55.820115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.820091    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:42:55.820115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.820104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-nk9qh\""
Sep 17 19:42:55.820313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.820167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-62pdx\""
Sep 17 19:42:55.820313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.820104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:42:55.820313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.820109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:42:55.827545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.827524    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:42:55.904382 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.904346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhg5s\" (UniqueName: \"kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:55.904573 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:55.904400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.005606 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.005572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhg5s\" (UniqueName: \"kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.005787 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.005623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.008376 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.008347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.013953 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.013931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhg5s\" (UniqueName: \"kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.128047 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.127959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:56.399802 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:56.399775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:42:56.402761 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:42:56.402733    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497f5aa3_b148_41c8_b716_4025da5ccc29.slice/crio-7039d641263c2e35b0db285ff480c7a315a2607320dec0d7c3ddeb2a8fac15ad WatchSource:0}: Error finding container 7039d641263c2e35b0db285ff480c7a315a2607320dec0d7c3ddeb2a8fac15ad: Status 404 returned error can't find the container with id 7039d641263c2e35b0db285ff480c7a315a2607320dec0d7c3ddeb2a8fac15ad
Sep 17 19:42:57.259583 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:57.259545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" event={"ID":"497f5aa3-b148-41c8-b716-4025da5ccc29","Type":"ContainerStarted","Data":"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"}
Sep 17 19:42:57.260084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:57.259622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" event={"ID":"497f5aa3-b148-41c8-b716-4025da5ccc29","Type":"ContainerStarted","Data":"7039d641263c2e35b0db285ff480c7a315a2607320dec0d7c3ddeb2a8fac15ad"}
Sep 17 19:42:57.260084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:57.259692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:42:57.276277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:42:57.276229    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podStartSLOduration=2.2762111369999998 podStartE2EDuration="2.276211137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:57.275350085 +0000 UTC m=+2209.288920257" watchObservedRunningTime="2026-09-17 19:42:57.276211137 +0000 UTC m=+2209.289781299"
Sep 17 19:43:28.265735 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:43:28.265655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:43:35.955595 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:43:35.955560    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:43:45.883682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:43:45.883643    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:43:50.975867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:43:50.975813    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main" containerID="cri-o://6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3" gracePeriod=45
Sep 17 19:43:55.883202 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:43:55.883170    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:44:05.883298 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:05.883247    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:44:05.883718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:05.883384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:44:15.883246 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:15.883208    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:44:21.224112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.224084    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:44:21.309936 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.309827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs\") pod \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") "
Sep 17 19:44:21.309936 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.309918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6j2\" (UniqueName: \"kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2\") pod \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\" (UID: \"2eed9cd1-f862-4158-bcdc-e8d7665e08ef\") "
Sep 17 19:44:21.312392 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.312353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2eed9cd1-f862-4158-bcdc-e8d7665e08ef" (UID: "2eed9cd1-f862-4158-bcdc-e8d7665e08ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:21.318867 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.312761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2" (OuterVolumeSpecName: "kube-api-access-9f6j2") pod "2eed9cd1-f862-4158-bcdc-e8d7665e08ef" (UID: "2eed9cd1-f862-4158-bcdc-e8d7665e08ef"). InnerVolumeSpecName "kube-api-access-9f6j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:21.411457 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.411391    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:44:21.411457 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.411452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6j2\" (UniqueName: \"kubernetes.io/projected/2eed9cd1-f862-4158-bcdc-e8d7665e08ef-kube-api-access-9f6j2\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:44:21.638904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.638807    2581 generic.go:356] "Generic (PLEG): container finished" podID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerID="6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3" exitCode=0
Sep 17 19:44:21.638904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.638868    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"
Sep 17 19:44:21.638904 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.638872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" event={"ID":"2eed9cd1-f862-4158-bcdc-e8d7665e08ef","Type":"ContainerDied","Data":"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"}
Sep 17 19:44:21.639190 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.638975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl" event={"ID":"2eed9cd1-f862-4158-bcdc-e8d7665e08ef","Type":"ContainerDied","Data":"01ad5d8746704e47c59e266668c8ae722da422b7a1885bff83292c9f436d6538"}
Sep 17 19:44:21.639190 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.639006    2581 scope.go:117] "RemoveContainer" containerID="6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"
Sep 17 19:44:21.648556 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.648538    2581 scope.go:117] "RemoveContainer" containerID="6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"
Sep 17 19:44:21.648827 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:44:21.648809    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3\": container with ID starting with 6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3 not found: ID does not exist" containerID="6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"
Sep 17 19:44:21.648868 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.648837    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3"} err="failed to get container status \"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3\": rpc error: code = NotFound desc = could not find container \"6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3\": container with ID starting with 6e818e120ca92cde7747679daa8fc3cbf27fe4608be574fc59201b48596a29d3 not found: ID does not exist"
Sep 17 19:44:21.659951 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.659925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:44:21.663197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:21.663176    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-qtvwl"]
Sep 17 19:44:22.138308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:44:22.138272    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" path="/var/lib/kubelet/pods/2eed9cd1-f862-4158-bcdc-e8d7665e08ef/volumes"
Sep 17 19:45:00.431130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:00.431096    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:45:08.265845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:08.265802    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:15.451301 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:15.451253    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main" containerID="cri-o://303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668" gracePeriod=45
Sep 17 19:45:18.265846 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:18.265795    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:28.265361 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:28.265327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:28.265835 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:28.265458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:45:38.266305 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:38.266270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:45.694175 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.694150    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:45:45.809325 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.809239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs\") pod \"497f5aa3-b148-41c8-b716-4025da5ccc29\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") "
Sep 17 19:45:45.809325 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.809282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhg5s\" (UniqueName: \"kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s\") pod \"497f5aa3-b148-41c8-b716-4025da5ccc29\" (UID: \"497f5aa3-b148-41c8-b716-4025da5ccc29\") "
Sep 17 19:45:45.811714 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.811688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s" (OuterVolumeSpecName: "kube-api-access-dhg5s") pod "497f5aa3-b148-41c8-b716-4025da5ccc29" (UID: "497f5aa3-b148-41c8-b716-4025da5ccc29"). InnerVolumeSpecName "kube-api-access-dhg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:45:45.811827 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.811727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "497f5aa3-b148-41c8-b716-4025da5ccc29" (UID: "497f5aa3-b148-41c8-b716-4025da5ccc29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:45:45.910319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.910269    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497f5aa3-b148-41c8-b716-4025da5ccc29-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:45:45.910319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:45.910314    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhg5s\" (UniqueName: \"kubernetes.io/projected/497f5aa3-b148-41c8-b716-4025da5ccc29-kube-api-access-dhg5s\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:45:46.023551 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.023511    2581 generic.go:356] "Generic (PLEG): container finished" podID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerID="303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668" exitCode=0
Sep 17 19:45:46.023717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.023563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"
Sep 17 19:45:46.023717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.023599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" event={"ID":"497f5aa3-b148-41c8-b716-4025da5ccc29","Type":"ContainerDied","Data":"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"}
Sep 17 19:45:46.023717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.023659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8" event={"ID":"497f5aa3-b148-41c8-b716-4025da5ccc29","Type":"ContainerDied","Data":"7039d641263c2e35b0db285ff480c7a315a2607320dec0d7c3ddeb2a8fac15ad"}
Sep 17 19:45:46.023717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.023682    2581 scope.go:117] "RemoveContainer" containerID="303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"
Sep 17 19:45:46.033429 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.033393    2581 scope.go:117] "RemoveContainer" containerID="303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"
Sep 17 19:45:46.033725 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:45:46.033705    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668\": container with ID starting with 303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668 not found: ID does not exist" containerID="303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"
Sep 17 19:45:46.033770 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.033734    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668"} err="failed to get container status \"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668\": rpc error: code = NotFound desc = could not find container \"303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668\": container with ID starting with 303a2d96cac9fd691f8cf9ec0e7adb4582ef57de793fcd1ab93420c52bd8c668 not found: ID does not exist"
Sep 17 19:45:46.044672 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.044647    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:45:46.048282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.048259    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58n5jd8"]
Sep 17 19:45:46.138117 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:46.138050    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" path="/var/lib/kubelet/pods/497f5aa3-b148-41c8-b716-4025da5ccc29/volumes"
Sep 17 19:45:50.787916 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.787887    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:45:50.788268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788191    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main"
Sep 17 19:45:50.788268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788202    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main"
Sep 17 19:45:50.788268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788216    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main"
Sep 17 19:45:50.788268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788221    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main"
Sep 17 19:45:50.788396 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eed9cd1-f862-4158-bcdc-e8d7665e08ef" containerName="main"
Sep 17 19:45:50.788396 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.788285    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="497f5aa3-b148-41c8-b716-4025da5ccc29" containerName="main"
Sep 17 19:45:50.792597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.792575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.796085 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.796065    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:45:50.796227 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.796087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:45:50.796390 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.796374    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-dwb45\""
Sep 17 19:45:50.796465 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.796434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:45:50.810715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.810685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:45:50.951333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951333 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951593 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6l9\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:50.951784 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:50.951614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052289 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052289 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052289 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6l9\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.052853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.053012 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.052985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.053055 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.053025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.053367 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.053351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.054838 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.054818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.055166 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.055148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.059977 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.059952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.060157 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.060140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6l9\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9\") pod \"router-gateway-1-openshift-default-5d8488d74d-vqgq5\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.102828 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.102808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:51.380939 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.380911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:45:51.383635 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:45:51.383613    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4bfbd3_da01_45b6_82cc_106c3daea286.slice/crio-91e9d604145995e1d853af580f9cb5b8550018b4cca687c60e5def8ef6435997 WatchSource:0}: Error finding container 91e9d604145995e1d853af580f9cb5b8550018b4cca687c60e5def8ef6435997: Status 404 returned error can't find the container with id 91e9d604145995e1d853af580f9cb5b8550018b4cca687c60e5def8ef6435997
Sep 17 19:45:51.386077 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.386043    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:45:51.386169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.386115    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:45:51.386169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:51.386147    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:45:52.055230 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:52.055188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerStarted","Data":"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"}
Sep 17 19:45:52.055636 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:52.055259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerStarted","Data":"91e9d604145995e1d853af580f9cb5b8550018b4cca687c60e5def8ef6435997"}
Sep 17 19:45:52.080468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:52.080393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podStartSLOduration=2.080376384 podStartE2EDuration="2.080376384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:45:52.079014887 +0000 UTC m=+2384.092585049" watchObservedRunningTime="2026-09-17 19:45:52.080376384 +0000 UTC m=+2384.093946586"
Sep 17 19:45:52.103915 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:52.103884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:52.105451 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:52.105428    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:45:53.107260 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:53.107229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:54.064442 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:54.064377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:45:54.153125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:45:54.153087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:46:03.106553 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.106518    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerID="754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc" exitCode=0
Sep 17 19:46:03.106994 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.106593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerDied","Data":"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"}
Sep 17 19:46:03.106994 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.106981    2581 scope.go:117] "RemoveContainer" containerID="754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"
Sep 17 19:46:03.108076 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.108049    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:46:03.108149 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.108123    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:46:03.108185 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:03.108160    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:46:04.114228 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:04.114193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerStarted","Data":"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"}
Sep 17 19:46:05.104084 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:05.104029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:46:05.105387 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:05.105354    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:46:06.103737 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:06.103689    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:46:07.104276 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:07.104231    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:46:08.104025 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:08.103985    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:46:09.064722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.064680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:46:09.103785 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.103744    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 17 19:46:09.447788 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.447752    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:46:09.450069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.450049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.452612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.452586    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 19:46:09.452863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.452847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4vt8m\""
Sep 17 19:46:09.453037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.453013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 19:46:09.453250 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.453233    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:46:09.467513 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.467489    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:46:09.512051 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcq9s\" (UniqueName: \"kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512200 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512404 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.512404 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.512262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcq9s\" (UniqueName: \"kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.613816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.613775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.614140 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.614098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.614215 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.614155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.614215 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.614200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.614379 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.614357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.614654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.614443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.616143 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.616123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.616475 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.616457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.624897 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.624877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcq9s\" (UniqueName: \"kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s\") pod \"router-with-refs-test-kserve-5b57c99d7-vdclb\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.705073 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.704993    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:46:09.707570 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.707553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.709889 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.709870    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-9wpp5\""
Sep 17 19:46:09.720073 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.720048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:46:09.760974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.760945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:09.815169 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.815121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrdr\" (UniqueName: \"kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.815311 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.815198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.916572 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.916533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrdr\" (UniqueName: \"kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.916773 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.916593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.919453 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.919403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:09.925268 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:09.925225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrdr\" (UniqueName: \"kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:10.017623 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.017550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:10.037659 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.037630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:46:10.039271 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:46:10.039237    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ac8134_0628_41b6_931f_8fb94af52959.slice/crio-3362e67f905bb8c865779ee551076ae83d8c4890fdef2fe152078746a5be1a19 WatchSource:0}: Error finding container 3362e67f905bb8c865779ee551076ae83d8c4890fdef2fe152078746a5be1a19: Status 404 returned error can't find the container with id 3362e67f905bb8c865779ee551076ae83d8c4890fdef2fe152078746a5be1a19
Sep 17 19:46:10.043400 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.043367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 19:46:10.108341 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.108312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:46:10.109617 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.109582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:46:10.142817 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.142771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerStarted","Data":"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"}
Sep 17 19:46:10.142974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.142831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerStarted","Data":"3362e67f905bb8c865779ee551076ae83d8c4890fdef2fe152078746a5be1a19"}
Sep 17 19:46:10.266574 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.266549    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:46:10.273747 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.273679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:46:10.295193 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:10.295157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:46:10.298619 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:46:10.298593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b76743_9466_43c5_a0ce_b2671579752b.slice/crio-5d83bbc8c0bd0312ed76605fff25dad4d9f4164a4010a7cc954eb8c972720d5a WatchSource:0}: Error finding container 5d83bbc8c0bd0312ed76605fff25dad4d9f4164a4010a7cc954eb8c972720d5a: Status 404 returned error can't find the container with id 5d83bbc8c0bd0312ed76605fff25dad4d9f4164a4010a7cc954eb8c972720d5a
Sep 17 19:46:11.151468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:11.151402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" event={"ID":"a9b76743-9466-43c5-a0ce-b2671579752b","Type":"ContainerStarted","Data":"59bf6bff4c0397486b3a826684f44a4b714be07cc0a5a19371d31b5c27274742"}
Sep 17 19:46:11.151989 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:11.151592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:11.151989 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:11.151649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" event={"ID":"a9b76743-9466-43c5-a0ce-b2671579752b","Type":"ContainerStarted","Data":"5d83bbc8c0bd0312ed76605fff25dad4d9f4164a4010a7cc954eb8c972720d5a"}
Sep 17 19:46:11.168597 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:11.168541    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podStartSLOduration=2.168523573 podStartE2EDuration="2.168523573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:11.166722779 +0000 UTC m=+2403.180292934" watchObservedRunningTime="2026-09-17 19:46:11.168523573 +0000 UTC m=+2403.182093770"
Sep 17 19:46:13.163069 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:13.163029    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4ac8134-0628-41b6-931f-8fb94af52959" containerID="e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220" exitCode=0
Sep 17 19:46:13.163472 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:13.163085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerDied","Data":"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"}
Sep 17 19:46:14.169756 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:14.169717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerStarted","Data":"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"}
Sep 17 19:46:14.192304 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:14.192224    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podStartSLOduration=2.9007339009999997 podStartE2EDuration="5.192202911s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.29146901s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:14.190598621 +0000 UTC m=+2406.204168818" watchObservedRunningTime="2026-09-17 19:46:14.192202911 +0000 UTC m=+2406.205773073"
Sep 17 19:46:14.761715 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:14.761676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:19.761720 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:19.761680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:46:19.763443 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:19.763388    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:46:29.761991 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:29.761948    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:46:39.762469 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:39.762391    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:46:42.158641 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:42.158610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:46:49.761977 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:49.761933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:46:59.762530 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:46:59.762475    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:09.762345 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:09.762296    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:19.761608 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:19.761567    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:29.761805 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:29.761760    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:39.761996 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:39.761952    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:49.761969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:49.761919    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:47:59.762356 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:47:59.762287    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 19:48:09.772339 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:09.772302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:48:09.780458 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:09.780391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:48:14.723463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:14.723409    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:48:14.727191 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:14.727162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:48:22.159174 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:22.159136    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:48:29.747583 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:29.747542    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main" containerID="cri-o://59bf6bff4c0397486b3a826684f44a4b714be07cc0a5a19371d31b5c27274742" gracePeriod=45
Sep 17 19:48:29.749324 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:29.749274    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main" containerID="cri-o://0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7" gracePeriod=45
Sep 17 19:48:30.496674 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.496647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:48:30.668541 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668541 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcq9s\" (UniqueName: \"kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.668794 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs\") pod \"d4ac8134-0628-41b6-931f-8fb94af52959\" (UID: \"d4ac8134-0628-41b6-931f-8fb94af52959\") "
Sep 17 19:48:30.669109 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.668935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:30.669109 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.669016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:48:30.669525 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.669355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home" (OuterVolumeSpecName: "home") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:30.669799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.669775    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.669799 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.669805    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac8134-0628-41b6-931f-8fb94af52959-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.669969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.669823    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.671166 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.671137    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm" (OuterVolumeSpecName: "dshm") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:30.671260 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.671223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s" (OuterVolumeSpecName: "kube-api-access-dcq9s") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "kube-api-access-dcq9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:48:30.672052 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.672032    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:48:30.680808 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.680764    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:30.698319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.698277    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4ac8134-0628-41b6-931f-8fb94af52959" (UID: "d4ac8134-0628-41b6-931f-8fb94af52959"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:30.771105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.771063    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ac8134-0628-41b6-931f-8fb94af52959-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.771105 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.771104    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcq9s\" (UniqueName: \"kubernetes.io/projected/d4ac8134-0628-41b6-931f-8fb94af52959-kube-api-access-dcq9s\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.771559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.771116    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.771559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.771125    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.771559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.771133    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ac8134-0628-41b6-931f-8fb94af52959-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:48:30.840534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.840499    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4ac8134-0628-41b6-931f-8fb94af52959" containerID="0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7" exitCode=0
Sep 17 19:48:30.840699 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.840555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"
Sep 17 19:48:30.840699 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.840558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerDied","Data":"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"}
Sep 17 19:48:30.840699 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.840693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb" event={"ID":"d4ac8134-0628-41b6-931f-8fb94af52959","Type":"ContainerDied","Data":"3362e67f905bb8c865779ee551076ae83d8c4890fdef2fe152078746a5be1a19"}
Sep 17 19:48:30.840817 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.840723    2581 scope.go:117] "RemoveContainer" containerID="0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"
Sep 17 19:48:30.849644 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.849626    2581 scope.go:117] "RemoveContainer" containerID="e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"
Sep 17 19:48:30.857853 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.857837    2581 scope.go:117] "RemoveContainer" containerID="0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"
Sep 17 19:48:30.858101 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:48:30.858082    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7\": container with ID starting with 0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7 not found: ID does not exist" containerID="0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"
Sep 17 19:48:30.858168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.858109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7"} err="failed to get container status \"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7\": rpc error: code = NotFound desc = could not find container \"0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7\": container with ID starting with 0bb33f57beb9268e37108b4f87b5ef68df62693e916da0a4103640e80c6b91e7 not found: ID does not exist"
Sep 17 19:48:30.858168 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.858127    2581 scope.go:117] "RemoveContainer" containerID="e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"
Sep 17 19:48:30.858387 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:48:30.858369    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220\": container with ID starting with e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220 not found: ID does not exist" containerID="e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"
Sep 17 19:48:30.858451 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.858393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220"} err="failed to get container status \"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220\": rpc error: code = NotFound desc = could not find container \"e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220\": container with ID starting with e2f2bd62cab49b308d4e8eb11fab60a1104fe1bb3e9f3b3f9a56e304697fc220 not found: ID does not exist"
Sep 17 19:48:30.863722 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.863700    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:48:30.868010 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:30.867985    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b57c99d7-vdclb"]
Sep 17 19:48:32.138566 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:32.138534    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" path="/var/lib/kubelet/pods/d4ac8134-0628-41b6-931f-8fb94af52959/volumes"
Sep 17 19:48:32.159027 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:32.158989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:48:42.160469 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:42.160328    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:48:42.161235 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:42.160672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:48:52.158389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:52.158343    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:48:59.971364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:59.971329    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9b76743-9466-43c5-a0ce-b2671579752b" containerID="59bf6bff4c0397486b3a826684f44a4b714be07cc0a5a19371d31b5c27274742" exitCode=0
Sep 17 19:48:59.971758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:59.971398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" event={"ID":"a9b76743-9466-43c5-a0ce-b2671579752b","Type":"ContainerDied","Data":"59bf6bff4c0397486b3a826684f44a4b714be07cc0a5a19371d31b5c27274742"}
Sep 17 19:48:59.996448 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:48:59.996407    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:49:00.117344 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.117220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs\") pod \"a9b76743-9466-43c5-a0ce-b2671579752b\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") "
Sep 17 19:49:00.117344 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.117314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlrdr\" (UniqueName: \"kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr\") pod \"a9b76743-9466-43c5-a0ce-b2671579752b\" (UID: \"a9b76743-9466-43c5-a0ce-b2671579752b\") "
Sep 17 19:49:00.119680 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.119648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr" (OuterVolumeSpecName: "kube-api-access-mlrdr") pod "a9b76743-9466-43c5-a0ce-b2671579752b" (UID: "a9b76743-9466-43c5-a0ce-b2671579752b"). InnerVolumeSpecName "kube-api-access-mlrdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:00.119782 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.119707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9b76743-9466-43c5-a0ce-b2671579752b" (UID: "a9b76743-9466-43c5-a0ce-b2671579752b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:49:00.218714 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.218667    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrdr\" (UniqueName: \"kubernetes.io/projected/a9b76743-9466-43c5-a0ce-b2671579752b-kube-api-access-mlrdr\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:00.218714 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.218705    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b76743-9466-43c5-a0ce-b2671579752b-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:00.976848 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.976812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5" event={"ID":"a9b76743-9466-43c5-a0ce-b2671579752b","Type":"ContainerDied","Data":"5d83bbc8c0bd0312ed76605fff25dad4d9f4164a4010a7cc954eb8c972720d5a"}
Sep 17 19:49:00.977306 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.976875    2581 scope.go:117] "RemoveContainer" containerID="59bf6bff4c0397486b3a826684f44a4b714be07cc0a5a19371d31b5c27274742"
Sep 17 19:49:00.977306 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.976832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"
Sep 17 19:49:00.995319 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:00.995290    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:49:01.003510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:01.003483    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-cs2l5"]
Sep 17 19:49:02.137683 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:02.137653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" path="/var/lib/kubelet/pods/a9b76743-9466-43c5-a0ce-b2671579752b/volumes"
Sep 17 19:49:09.712171 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:09.712128    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:49:09.712784 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:09.712354    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" containerID="cri-o://0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf" gracePeriod=30
Sep 17 19:49:10.110989 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:10.110900    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:49:12.534811 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.534777    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535095    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535106    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535119    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="storage-initializer"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535124    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="storage-initializer"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535143    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535149    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main"
Sep 17 19:49:12.535195 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535198    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b76743-9466-43c5-a0ce-b2671579752b" containerName="main"
Sep 17 19:49:12.535433 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.535206    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ac8134-0628-41b6-931f-8fb94af52959" containerName="main"
Sep 17 19:49:12.539344 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.539325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.541858 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.541833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:49:12.541992 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.541928    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:49:12.541992 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.541972    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-2c48r\""
Sep 17 19:49:12.542112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.541987    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:49:12.542112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.542024    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:49:12.542112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.541977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 19:49:12.542241 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.542130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-w655m\""
Sep 17 19:49:12.552380 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.552358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:49:12.629163 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629354 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629354 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqxv\" (UniqueName: \"kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629354 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.629505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.629476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730353 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730353 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqxv\" (UniqueName: \"kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730611 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730839 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730896 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.730956 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.731016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.730979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.731092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.731067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.731198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.731181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.733080 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.733064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.733371 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.733350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.739230 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.739203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqxv\" (UniqueName: \"kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.849021 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.848912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:12.868483 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.868450    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:49:12.872497 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.872471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:12.875074 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.875049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-ks85m\""
Sep 17 19:49:12.881673 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:12.881639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:49:13.033481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.033444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.033654 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.033531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfh5\" (UniqueName: \"kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.134528 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.134393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.134718 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.134542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfh5\" (UniqueName: \"kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.137941 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.137901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.139744 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.139697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:49:13.143568 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:49:13.143536    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0db1f6_10be_4ac5_a4f1_c9d43c9e3474.slice/crio-cac54f23306bad3f14791be3921024be41a99a36927f087aa7f121a1fc59c144 WatchSource:0}: Error finding container cac54f23306bad3f14791be3921024be41a99a36927f087aa7f121a1fc59c144: Status 404 returned error can't find the container with id cac54f23306bad3f14791be3921024be41a99a36927f087aa7f121a1fc59c144
Sep 17 19:49:13.143814 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.143798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfh5\" (UniqueName: \"kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.190108 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.190083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:13.468000 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:13.467958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:49:13.471155 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:49:13.471122    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b083758_dd33_41f2_b0dd_47b2a789c2ab.slice/crio-e2f8fb31b97a247169926d080406954ca59f69874b0cf11b73743d7366fea588 WatchSource:0}: Error finding container e2f8fb31b97a247169926d080406954ca59f69874b0cf11b73743d7366fea588: Status 404 returned error can't find the container with id e2f8fb31b97a247169926d080406954ca59f69874b0cf11b73743d7366fea588
Sep 17 19:49:14.039183 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.039141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" event={"ID":"6b083758-dd33-41f2-b0dd-47b2a789c2ab","Type":"ContainerStarted","Data":"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"}
Sep 17 19:49:14.039673 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.039200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" event={"ID":"6b083758-dd33-41f2-b0dd-47b2a789c2ab","Type":"ContainerStarted","Data":"e2f8fb31b97a247169926d080406954ca59f69874b0cf11b73743d7366fea588"}
Sep 17 19:49:14.039673 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.039252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:14.040956 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.040935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerStarted","Data":"9c157b1c7c1a460bf10397ea3d63c9c63157de31f6ea95bf63791a116680c835"}
Sep 17 19:49:14.041055 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.040973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerStarted","Data":"cac54f23306bad3f14791be3921024be41a99a36927f087aa7f121a1fc59c144"}
Sep 17 19:49:14.041055 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.041017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:14.056803 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.056757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podStartSLOduration=2.056741598 podStartE2EDuration="2.056741598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:49:14.055243202 +0000 UTC m=+2586.068813364" watchObservedRunningTime="2026-09-17 19:49:14.056741598 +0000 UTC m=+2586.070311759"
Sep 17 19:49:14.970491 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:14.970461    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:49:15.052524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.051879    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerID="0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf" exitCode=0
Sep 17 19:49:15.052524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.051968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerDied","Data":"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"}
Sep 17 19:49:15.052524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5" event={"ID":"bf4bfbd3-da01-45b6-82cc-106c3daea286","Type":"ContainerDied","Data":"91e9d604145995e1d853af580f9cb5b8550018b4cca687c60e5def8ef6435997"}
Sep 17 19:49:15.052524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052057    2581 scope.go:117] "RemoveContainer" containerID="0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"
Sep 17 19:49:15.052524 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj6l9\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.052954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo\") pod \"bf4bfbd3-da01-45b6-82cc-106c3daea286\" (UID: \"bf4bfbd3-da01-45b6-82cc-106c3daea286\") "
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:15.053256 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:15.053863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053293    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-data\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.053863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.053310    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.055612    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.055776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.056024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.056544    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.057017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:49:15.058428 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.057441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9" (OuterVolumeSpecName: "kube-api-access-gj6l9") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "kube-api-access-gj6l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:15.061305 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.061121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerStarted","Data":"8bd2261ddb337aa2a6d2ffd2397b145f26a2174690a738d1d8898b8255aa76b3"}
Sep 17 19:49:15.061820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.061796    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf4bfbd3-da01-45b6-82cc-106c3daea286" (UID: "bf4bfbd3-da01-45b6-82cc-106c3daea286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:15.078822 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.078793    2581 scope.go:117] "RemoveContainer" containerID="754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"
Sep 17 19:49:15.087558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.087518    2581 scope.go:117] "RemoveContainer" containerID="0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"
Sep 17 19:49:15.087837 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:49:15.087808    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf\": container with ID starting with 0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf not found: ID does not exist" containerID="0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"
Sep 17 19:49:15.087909 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.087850    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf"} err="failed to get container status \"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf\": rpc error: code = NotFound desc = could not find container \"0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf\": container with ID starting with 0e71a9277f3eda6fcfe01ee86debc6a3d972b340506e615fd62ab80c16daeebf not found: ID does not exist"
Sep 17 19:49:15.087909 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.087874    2581 scope.go:117] "RemoveContainer" containerID="754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"
Sep 17 19:49:15.088179 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:49:15.088162    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc\": container with ID starting with 754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc not found: ID does not exist" containerID="754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"
Sep 17 19:49:15.088244 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.088183    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc"} err="failed to get container status \"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc\": rpc error: code = NotFound desc = could not find container \"754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc\": container with ID starting with 754ef6761f39ab1cf27810a701fedd1d39808a5195d633410764b88b52d699fc not found: ID does not exist"
Sep 17 19:49:15.154296 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154235    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-workload-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154360    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4bfbd3-da01-45b6-82cc-106c3daea286-istiod-ca-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154386    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-token\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154396    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-credential-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154405    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-podinfo\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154437    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4bfbd3-da01-45b6-82cc-106c3daea286-istio-envoy\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.154544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.154453    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6l9\" (UniqueName: \"kubernetes.io/projected/bf4bfbd3-da01-45b6-82cc-106c3daea286-kube-api-access-gj6l9\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:49:15.374857 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.374822    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:49:15.381456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:15.381404    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-vqgq5"]
Sep 17 19:49:16.138845 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:16.138811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" path="/var/lib/kubelet/pods/bf4bfbd3-da01-45b6-82cc-106c3daea286/volumes"
Sep 17 19:49:17.078132 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:17.078098    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerID="8bd2261ddb337aa2a6d2ffd2397b145f26a2174690a738d1d8898b8255aa76b3" exitCode=0
Sep 17 19:49:17.078332 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:17.078167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerDied","Data":"8bd2261ddb337aa2a6d2ffd2397b145f26a2174690a738d1d8898b8255aa76b3"}
Sep 17 19:49:18.086051 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:18.086009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerStarted","Data":"6fdbc684dc5788a02c30095b1cb12374ee621190b1440b1dd8f4d6e16cb89c77"}
Sep 17 19:49:18.111394 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:18.111324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podStartSLOduration=3.684779082 podStartE2EDuration="6.111300997s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.426521915s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:49:18.109055502 +0000 UTC m=+2590.122625698" watchObservedRunningTime="2026-09-17 19:49:18.111300997 +0000 UTC m=+2590.124871159"
Sep 17 19:49:22.850036 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:22.849923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:22.850036 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:22.850018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:22.851713 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:22.851678    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:49:32.850345 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:32.850290    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:49:32.865655 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:32.865572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:49:42.850132 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:42.850089    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:49:45.062959 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:45.062927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:49:52.849603 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:49:52.849555    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:02.850478 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:02.850391    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:12.849921 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:12.849872    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:22.849863 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:22.849819    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:32.850369 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:32.850315    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:42.850399 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:42.850341    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:50:52.849495 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:50:52.849376    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:51:02.849933 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:02.849884    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:51:10.298841 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:10.298809    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:51:10.306578 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:10.306545    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:51:12.849969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:12.849914    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8001/health\": dial tcp 10.132.0.60:8001: connect: connection refused"
Sep 17 19:51:22.865739 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:22.865702    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:51:22.877681 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:22.877653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:51:34.042115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:34.042076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:51:34.043698 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:34.043665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:51:35.062517 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:35.062476    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:51:45.061912 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:45.061871    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:51:49.063511 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.063465    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" containerID="cri-o://e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539" gracePeriod=45
Sep 17 19:51:49.066038 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.066004    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main" containerID="cri-o://6fdbc684dc5788a02c30095b1cb12374ee621190b1440b1dd8f4d6e16cb89c77" gracePeriod=45
Sep 17 19:51:49.684125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.684082    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="llm-d-routing-sidecar" containerID="cri-o://9c157b1c7c1a460bf10397ea3d63c9c63157de31f6ea95bf63791a116680c835" gracePeriod=45
Sep 17 19:51:49.847220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.847188    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerID="6fdbc684dc5788a02c30095b1cb12374ee621190b1440b1dd8f4d6e16cb89c77" exitCode=0
Sep 17 19:51:49.847220 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.847215    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerID="9c157b1c7c1a460bf10397ea3d63c9c63157de31f6ea95bf63791a116680c835" exitCode=0
Sep 17 19:51:49.847468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.847261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerDied","Data":"6fdbc684dc5788a02c30095b1cb12374ee621190b1440b1dd8f4d6e16cb89c77"}
Sep 17 19:51:49.847468 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.847328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerDied","Data":"9c157b1c7c1a460bf10397ea3d63c9c63157de31f6ea95bf63791a116680c835"}
Sep 17 19:51:49.941203 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:49.941178    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:51:50.034719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lqxv\" (UniqueName: \"kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.034930 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.034898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm\") pod \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\" (UID: \"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474\") "
Sep 17 19:51:50.035292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.035029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.035292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.035197    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.035292 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.035255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:50.035734 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.035661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home" (OuterVolumeSpecName: "home") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.037646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.037616    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv" (OuterVolumeSpecName: "kube-api-access-5lqxv") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "kube-api-access-5lqxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:51:50.037646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.037636    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm" (OuterVolumeSpecName: "dshm") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.037829 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.037800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:51:50.048473 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.048405    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.060313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.060274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" (UID: "cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.135973    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136021    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lqxv\" (UniqueName: \"kubernetes.io/projected/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kube-api-access-5lqxv\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136056    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136074    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136094    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.140523 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.136109    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.853360 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.853322    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"
Sep 17 19:51:50.853568 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.853315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s" event={"ID":"cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474","Type":"ContainerDied","Data":"cac54f23306bad3f14791be3921024be41a99a36927f087aa7f121a1fc59c144"}
Sep 17 19:51:50.853568 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.853467    2581 scope.go:117] "RemoveContainer" containerID="6fdbc684dc5788a02c30095b1cb12374ee621190b1440b1dd8f4d6e16cb89c77"
Sep 17 19:51:50.863539 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.863485    2581 scope.go:117] "RemoveContainer" containerID="8bd2261ddb337aa2a6d2ffd2397b145f26a2174690a738d1d8898b8255aa76b3"
Sep 17 19:51:50.872364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.872344    2581 scope.go:117] "RemoveContainer" containerID="9c157b1c7c1a460bf10397ea3d63c9c63157de31f6ea95bf63791a116680c835"
Sep 17 19:51:50.873154 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.873124    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:51:50.879519 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:50.879489    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-78d6b894fcb467s"]
Sep 17 19:51:52.137666 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:52.137627    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" path="/var/lib/kubelet/pods/cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474/volumes"
Sep 17 19:51:55.062627 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:55.062587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:51:55.063006 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:51:55.062705    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:52:05.062383 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:05.062344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:52:15.062922 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:15.062884    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:52:19.311754 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.311722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:52:19.357661 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.357572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpfh5\" (UniqueName: \"kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5\") pod \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") "
Sep 17 19:52:19.357826 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.357666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs\") pod \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\" (UID: \"6b083758-dd33-41f2-b0dd-47b2a789c2ab\") "
Sep 17 19:52:19.360041 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.360004    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5" (OuterVolumeSpecName: "kube-api-access-kpfh5") pod "6b083758-dd33-41f2-b0dd-47b2a789c2ab" (UID: "6b083758-dd33-41f2-b0dd-47b2a789c2ab"). InnerVolumeSpecName "kube-api-access-kpfh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:52:19.360182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.360127    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b083758-dd33-41f2-b0dd-47b2a789c2ab" (UID: "6b083758-dd33-41f2-b0dd-47b2a789c2ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:52:19.458775 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.458726    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b083758-dd33-41f2-b0dd-47b2a789c2ab-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:52:19.458775 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.458765    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpfh5\" (UniqueName: \"kubernetes.io/projected/6b083758-dd33-41f2-b0dd-47b2a789c2ab-kube-api-access-kpfh5\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:52:19.984271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.984178    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerID="e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539" exitCode=0
Sep 17 19:52:19.984271 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.984235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"
Sep 17 19:52:19.984550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.984268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" event={"ID":"6b083758-dd33-41f2-b0dd-47b2a789c2ab","Type":"ContainerDied","Data":"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"}
Sep 17 19:52:19.984550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.984367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs" event={"ID":"6b083758-dd33-41f2-b0dd-47b2a789c2ab","Type":"ContainerDied","Data":"e2f8fb31b97a247169926d080406954ca59f69874b0cf11b73743d7366fea588"}
Sep 17 19:52:19.984550 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.984398    2581 scope.go:117] "RemoveContainer" containerID="e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"
Sep 17 19:52:19.994700 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.994678    2581 scope.go:117] "RemoveContainer" containerID="e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"
Sep 17 19:52:19.995068 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:52:19.995047    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539\": container with ID starting with e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539 not found: ID does not exist" containerID="e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"
Sep 17 19:52:19.995138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:19.995078    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539"} err="failed to get container status \"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539\": rpc error: code = NotFound desc = could not find container \"e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539\": container with ID starting with e84712c12f7981346a7a60bcd4e8c23ce3f5e8b8ad80166682d2b26ed1d3f539 not found: ID does not exist"
Sep 17 19:52:20.008312 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:20.008275    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:52:20.014496 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:20.014466    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6hqjs"]
Sep 17 19:52:20.136857 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:20.136826    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" path="/var/lib/kubelet/pods/6b083758-dd33-41f2-b0dd-47b2a789c2ab/volumes"
Sep 17 19:52:31.523368 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523329    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523689    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523703    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523719    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="storage-initializer"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523725    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="storage-initializer"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="llm-d-routing-sidecar"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523740    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="llm-d-routing-sidecar"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523747    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523752    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main"
Sep 17 19:52:31.523798 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523767    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523822    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523832    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523843    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="main"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523851    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0db1f6-10be-4ac5-a4f1-c9d43c9e3474" containerName="llm-d-routing-sidecar"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523857    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b083758-dd33-41f2-b0dd-47b2a789c2ab" containerName="main"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523913    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.524125 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.523919    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4bfbd3-da01-45b6-82cc-106c3daea286" containerName="istio-proxy"
Sep 17 19:52:31.528714 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.528690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.531335 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.531307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:52:31.531620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.531439    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:52:31.531811 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.531792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cn4l7\""
Sep 17 19:52:31.532088 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.532063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:52:31.532211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.532126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:52:31.532211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.532149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-rtzhx\""
Sep 17 19:52:31.532211 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.532174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:52:31.547050 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.547017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:52:31.663761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.663761 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb22r\" (UniqueName: \"kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.663974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.663974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.663974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.663974 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.664112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.663983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.664112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.664011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765092 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb22r\" (UniqueName: \"kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765366 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765787 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765886 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.765955 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.765939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.766277 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.766256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.767856 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.767838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.768181 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.768153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.776496 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.776408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb22r\" (UniqueName: \"kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r\") pod \"custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.806380 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.806344    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:52:31.810658 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.810635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.813135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.813111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-tt9xc\""
Sep 17 19:52:31.818280 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.818256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:52:31.839027 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.838983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:31.866538 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.866501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfnv\" (UniqueName: \"kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.866697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.866607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.967910 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.967868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfnv\" (UniqueName: \"kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.968091 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.967929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.970945 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.970917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:31.977020 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:31.976988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfnv\" (UniqueName: \"kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:32.121463 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:32.121260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:52:32.122376 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:32.122351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:32.124336 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:52:32.124309    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14479dcc_2089_4052_8498_32d47e49f512.slice/crio-a28770be950de4209e126f72bd7aa509c9f360866f2339dc2fa0711b39fd3469 WatchSource:0}: Error finding container a28770be950de4209e126f72bd7aa509c9f360866f2339dc2fa0711b39fd3469: Status 404 returned error can't find the container with id a28770be950de4209e126f72bd7aa509c9f360866f2339dc2fa0711b39fd3469
Sep 17 19:52:32.402922 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:32.402838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:52:32.406374 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:52:32.406332    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8042e8e3_e8f4_45a5_b461_d524c3a1514e.slice/crio-968a37f740708775a6dec97eddf267fa95f5774d6e614b66f99a784edc673d2e WatchSource:0}: Error finding container 968a37f740708775a6dec97eddf267fa95f5774d6e614b66f99a784edc673d2e: Status 404 returned error can't find the container with id 968a37f740708775a6dec97eddf267fa95f5774d6e614b66f99a784edc673d2e
Sep 17 19:52:33.046970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.046923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" event={"ID":"8042e8e3-e8f4-45a5-b461-d524c3a1514e","Type":"ContainerStarted","Data":"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"}
Sep 17 19:52:33.047439 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.047002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:52:33.047439 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.047027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" event={"ID":"8042e8e3-e8f4-45a5-b461-d524c3a1514e","Type":"ContainerStarted","Data":"968a37f740708775a6dec97eddf267fa95f5774d6e614b66f99a784edc673d2e"}
Sep 17 19:52:33.048548 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.048528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerStarted","Data":"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"}
Sep 17 19:52:33.048621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.048572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerStarted","Data":"a28770be950de4209e126f72bd7aa509c9f360866f2339dc2fa0711b39fd3469"}
Sep 17 19:52:33.048621 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.048605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:33.064330 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:33.064267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podStartSLOduration=2.064245173 podStartE2EDuration="2.064245173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:33.063916702 +0000 UTC m=+2785.077486864" watchObservedRunningTime="2026-09-17 19:52:33.064245173 +0000 UTC m=+2785.077815335"
Sep 17 19:52:34.058116 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:34.058070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerStarted","Data":"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"}
Sep 17 19:52:36.072151 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:36.072104    2581 generic.go:356] "Generic (PLEG): container finished" podID="14479dcc-2089-4052-8498-32d47e49f512" containerID="a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355" exitCode=0
Sep 17 19:52:36.072558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:36.072185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerDied","Data":"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"}
Sep 17 19:52:37.083152 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:37.083106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerStarted","Data":"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"}
Sep 17 19:52:37.118179 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:37.118123    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podStartSLOduration=3.80581973 podStartE2EDuration="6.118105694s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.312285964s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:37.117763256 +0000 UTC m=+2789.131333417" watchObservedRunningTime="2026-09-17 19:52:37.118105694 +0000 UTC m=+2789.131675855"
Sep 17 19:52:41.839216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:41.839170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:41.839656 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:41.839271    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:52:41.840432 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:41.840391    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:52:51.839734 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:51.839680    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:52:51.857516 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:52:51.857484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:53:01.839831 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:01.839781    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:53:04.059858 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:04.059822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:53:11.839876 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:11.839813    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:53:21.840382 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:21.840319    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:53:31.839826 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:31.839773    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:53:41.840559 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:41.840500    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:53:51.839970 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:53:51.839859    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:54:01.840135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:01.840087    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:54:11.840136 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:11.840088    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:54:21.839995 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:21.839934    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:54:31.840356 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:31.840304    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Sep 17 19:54:41.850100 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:41.850061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:54:41.867992 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:41.867957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:54:54.420483 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:54.420406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:54:54.426693 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:54:54.426660    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:55:04.059767 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:04.059719    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:55:09.446389 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:09.446318    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main" containerID="cri-o://f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c" gracePeriod=45
Sep 17 19:55:09.455762 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:09.455722    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main" containerID="cri-o://de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f" gracePeriod=45
Sep 17 19:55:10.067368 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.067333    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="llm-d-routing-sidecar" containerID="cri-o://64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393" gracePeriod=45
Sep 17 19:55:10.325844 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.325815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:55:10.494352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494352 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb22r\" (UniqueName: \"kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm\") pod \"14479dcc-2089-4052-8498-32d47e49f512\" (UID: \"14479dcc-2089-4052-8498-32d47e49f512\") "
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache" (OuterVolumeSpecName: "model-cache") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494907    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14479dcc-2089-4052-8498-32d47e49f512-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.494950 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.494927    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.495656 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.495237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home" (OuterVolumeSpecName: "home") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.497225 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.497195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:10.497225 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.497199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r" (OuterVolumeSpecName: "kube-api-access-bb22r") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "kube-api-access-bb22r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:10.497446 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.497303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm" (OuterVolumeSpecName: "dshm") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.507499 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.507457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.521002 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.520962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14479dcc-2089-4052-8498-32d47e49f512" (UID: "14479dcc-2089-4052-8498-32d47e49f512"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595782    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595815    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595825    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14479dcc-2089-4052-8498-32d47e49f512-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595835    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595845    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14479dcc-2089-4052-8498-32d47e49f512-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.595879 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.595854    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb22r\" (UniqueName: \"kubernetes.io/projected/14479dcc-2089-4052-8498-32d47e49f512-kube-api-access-bb22r\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.831112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831072    2581 generic.go:356] "Generic (PLEG): container finished" podID="14479dcc-2089-4052-8498-32d47e49f512" containerID="de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f" exitCode=0
Sep 17 19:55:10.831112 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831104    2581 generic.go:356] "Generic (PLEG): container finished" podID="14479dcc-2089-4052-8498-32d47e49f512" containerID="64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393" exitCode=0
Sep 17 19:55:10.831350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831130    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"
Sep 17 19:55:10.831350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerDied","Data":"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"}
Sep 17 19:55:10.831350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerDied","Data":"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"}
Sep 17 19:55:10.831350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd" event={"ID":"14479dcc-2089-4052-8498-32d47e49f512","Type":"ContainerDied","Data":"a28770be950de4209e126f72bd7aa509c9f360866f2339dc2fa0711b39fd3469"}
Sep 17 19:55:10.831350 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.831243    2581 scope.go:117] "RemoveContainer" containerID="de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"
Sep 17 19:55:10.840632 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.840610    2581 scope.go:117] "RemoveContainer" containerID="a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"
Sep 17 19:55:10.848946 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.848925    2581 scope.go:117] "RemoveContainer" containerID="64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"
Sep 17 19:55:10.856700 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.856680    2581 scope.go:117] "RemoveContainer" containerID="de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"
Sep 17 19:55:10.856979 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:55:10.856961    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f\": container with ID starting with de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f not found: ID does not exist" containerID="de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"
Sep 17 19:55:10.857026 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.856988    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"} err="failed to get container status \"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f\": rpc error: code = NotFound desc = could not find container \"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f\": container with ID starting with de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f not found: ID does not exist"
Sep 17 19:55:10.857026 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857006    2581 scope.go:117] "RemoveContainer" containerID="a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"
Sep 17 19:55:10.857269 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:55:10.857253    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355\": container with ID starting with a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355 not found: ID does not exist" containerID="a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"
Sep 17 19:55:10.857313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857273    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"} err="failed to get container status \"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355\": rpc error: code = NotFound desc = could not find container \"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355\": container with ID starting with a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355 not found: ID does not exist"
Sep 17 19:55:10.857313 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857286    2581 scope.go:117] "RemoveContainer" containerID="64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"
Sep 17 19:55:10.857551 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:55:10.857533    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393\": container with ID starting with 64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393 not found: ID does not exist" containerID="64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"
Sep 17 19:55:10.857612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857556    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"} err="failed to get container status \"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393\": rpc error: code = NotFound desc = could not find container \"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393\": container with ID starting with 64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393 not found: ID does not exist"
Sep 17 19:55:10.857612 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857569    2581 scope.go:117] "RemoveContainer" containerID="de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"
Sep 17 19:55:10.857821 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857797    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f"} err="failed to get container status \"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f\": rpc error: code = NotFound desc = could not find container \"de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f\": container with ID starting with de3c13908458420e752d127f64ef4010ed320358459a5c2e5538effbabc8a35f not found: ID does not exist"
Sep 17 19:55:10.857865 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.857827    2581 scope.go:117] "RemoveContainer" containerID="a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"
Sep 17 19:55:10.858060 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.858041    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355"} err="failed to get container status \"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355\": rpc error: code = NotFound desc = could not find container \"a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355\": container with ID starting with a0671b62ff30bbb843e2d6c95edc2d1c53c21ad4f564b81dfad04050d6ef3355 not found: ID does not exist"
Sep 17 19:55:10.858100 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.858060    2581 scope.go:117] "RemoveContainer" containerID="64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"
Sep 17 19:55:10.858269 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.858254    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393"} err="failed to get container status \"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393\": rpc error: code = NotFound desc = could not find container \"64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393\": container with ID starting with 64061d3a609c99f38181470913391d742696242f1f8e60ace9fa4c931d296393 not found: ID does not exist"
Sep 17 19:55:10.880138 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.880098    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:55:10.897598 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:10.897553    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76756ccccb-xl5rd"]
Sep 17 19:55:12.137369 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:12.137333    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14479dcc-2089-4052-8498-32d47e49f512" path="/var/lib/kubelet/pods/14479dcc-2089-4052-8498-32d47e49f512/volumes"
Sep 17 19:55:14.060239 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:14.060178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:55:24.060985 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:24.060884    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:55:24.061774 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:24.061750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:55:34.060393 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:34.060347    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:55:39.701884 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.701853    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:55:39.831364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.831272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtfnv\" (UniqueName: \"kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv\") pod \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") "
Sep 17 19:55:39.831364 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.831313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs\") pod \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\" (UID: \"8042e8e3-e8f4-45a5-b461-d524c3a1514e\") "
Sep 17 19:55:39.833713 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.833682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8042e8e3-e8f4-45a5-b461-d524c3a1514e" (UID: "8042e8e3-e8f4-45a5-b461-d524c3a1514e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:39.833713 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.833690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv" (OuterVolumeSpecName: "kube-api-access-dtfnv") pod "8042e8e3-e8f4-45a5-b461-d524c3a1514e" (UID: "8042e8e3-e8f4-45a5-b461-d524c3a1514e"). InnerVolumeSpecName "kube-api-access-dtfnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:39.932434 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.932382    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfnv\" (UniqueName: \"kubernetes.io/projected/8042e8e3-e8f4-45a5-b461-d524c3a1514e-kube-api-access-dtfnv\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:39.932614 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.932454    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042e8e3-e8f4-45a5-b461-d524c3a1514e-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:55:39.966576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.966542    2581 generic.go:356] "Generic (PLEG): container finished" podID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerID="f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c" exitCode=0
Sep 17 19:55:39.966748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.966590    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"
Sep 17 19:55:39.966748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.966616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" event={"ID":"8042e8e3-e8f4-45a5-b461-d524c3a1514e","Type":"ContainerDied","Data":"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"}
Sep 17 19:55:39.966748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.966659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t" event={"ID":"8042e8e3-e8f4-45a5-b461-d524c3a1514e","Type":"ContainerDied","Data":"968a37f740708775a6dec97eddf267fa95f5774d6e614b66f99a784edc673d2e"}
Sep 17 19:55:39.966748 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.966680    2581 scope.go:117] "RemoveContainer" containerID="f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"
Sep 17 19:55:39.976609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.976589    2581 scope.go:117] "RemoveContainer" containerID="f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"
Sep 17 19:55:39.976929 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:55:39.976906    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c\": container with ID starting with f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c not found: ID does not exist" containerID="f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"
Sep 17 19:55:39.976978 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.976940    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c"} err="failed to get container status \"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c\": rpc error: code = NotFound desc = could not find container \"f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c\": container with ID starting with f2fdc35b06f1b09df32843b1f93c915baf057fbe1e4f6dd8280f8f5acb9b280c not found: ID does not exist"
Sep 17 19:55:39.987785 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.987755    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:55:39.991609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:39.991582    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48n4b5t"]
Sep 17 19:55:40.137124 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:40.137039    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" path="/var/lib/kubelet/pods/8042e8e3-e8f4-45a5-b461-d524c3a1514e/volumes"
Sep 17 19:55:44.981646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981613    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981936    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981947    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981966    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="llm-d-routing-sidecar"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981973    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="llm-d-routing-sidecar"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981983    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="storage-initializer"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="storage-initializer"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.981997    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main"
Sep 17 19:55:44.982016 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.982002    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main"
Sep 17 19:55:44.982267 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.982047    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8042e8e3-e8f4-45a5-b461-d524c3a1514e" containerName="main"
Sep 17 19:55:44.982267 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.982056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="llm-d-routing-sidecar"
Sep 17 19:55:44.982267 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.982062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="14479dcc-2089-4052-8498-32d47e49f512" containerName="main"
Sep 17 19:55:44.986198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.986174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:44.989941 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.989909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:55:44.991741 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.990283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-gmz87\""
Sep 17 19:55:44.991741 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.990326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:55:44.991741 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:44.991203    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:55:45.005996 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.005968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 19:55:45.077242 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077242 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.077504 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.077499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhb4\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178777 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.178969 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.178902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhb4\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179243 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179456 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179590 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179590 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179590 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.179709 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.179676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.181395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.181374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.181646 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.181627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.187212 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.187180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.187356 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.187315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhb4\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4\") pod \"router-gateway-2-openshift-default-57b79bd49b-qwb6k\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.304811 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.304703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:45.585055 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.584974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 19:55:45.589012 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:55:45.588980    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e33a32_a294_447e_bea0_2b2ce6e884ba.slice/crio-e8500269e265cb22248a94ae1c375ab3a5de665b4f1c4ebd6469ef1bb32a7b7d WatchSource:0}: Error finding container e8500269e265cb22248a94ae1c375ab3a5de665b4f1c4ebd6469ef1bb32a7b7d: Status 404 returned error can't find the container with id e8500269e265cb22248a94ae1c375ab3a5de665b4f1c4ebd6469ef1bb32a7b7d
Sep 17 19:55:45.591127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.591088    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:45.591217 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.591201    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:45.591254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:45.591240    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:46.000717 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:46.000667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerStarted","Data":"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"}
Sep 17 19:55:46.001127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:46.000732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerStarted","Data":"e8500269e265cb22248a94ae1c375ab3a5de665b4f1c4ebd6469ef1bb32a7b7d"}
Sep 17 19:55:46.025803 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:46.025743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podStartSLOduration=2.025722707 podStartE2EDuration="2.025722707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:55:46.024141022 +0000 UTC m=+2978.037711182" watchObservedRunningTime="2026-09-17 19:55:46.025722707 +0000 UTC m=+2978.039292869"
Sep 17 19:55:46.305261 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:46.305154    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:46.310696 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:46.310664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:47.005634 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:47.005598    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:47.006702 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:47.006674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:58.061710 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.061676    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerID="828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897" exitCode=0
Sep 17 19:55:58.062135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.061722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerDied","Data":"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"}
Sep 17 19:55:58.062135 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.062107    2581 scope.go:117] "RemoveContainer" containerID="828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"
Sep 17 19:55:58.063146 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.063116    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:58.063218 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.063190    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:58.063261 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:58.063230    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 19:55:59.068545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:59.068501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerStarted","Data":"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"}
Sep 17 19:55:59.305908 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:59.305857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:55:59.307088 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:55:59.307057    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 19:56:00.305231 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:00.305186    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 19:56:01.305501 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:01.305454    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 19:56:02.006308 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:02.006258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:56:02.305826 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:02.305722    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 19:56:03.305181 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:03.305137    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 19:56:04.309724 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:04.309690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:56:04.310896 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:04.310876    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 19:56:10.334182 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:10.334147    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:56:10.343098 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:10.343072    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 19:56:42.231053 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.231011    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:56:42.234544 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.234522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.238839 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.238812    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:56:42.239067 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.239044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-7zfg2\""
Sep 17 19:56:42.239130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.239049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:56:42.239130 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.239092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:56:42.239742 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.239724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-nmg8w\""
Sep 17 19:56:42.257259 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.257225    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:56:42.382784 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.382784 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2dj\" (UniqueName: \"kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.382958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.383006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.383037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.383031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.483820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.483718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.483820 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.483775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.483825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.483853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2dj\" (UniqueName: \"kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484042 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484216 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484323 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484479 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484588 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484588 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.484779 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.484646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.486388 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.486354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.486877 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.486860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.493813 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.493764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2dj\" (UniqueName: \"kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj\") pod \"router-with-refs-pd-test-kserve-599dcc896b-h48m2\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.543964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.543924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:42.747176 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.747078    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:56:42.750656 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.750634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:42.754656 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.754627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-95rnb\""
Sep 17 19:56:42.766230 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.766198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:56:42.828615 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.828585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:56:42.831595 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:56:42.831557    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677b9a11_9959_40ee_803c_5a062fe1e468.slice/crio-cd3f7ce1c69aa6c67c22365c06fa791b404e58f582d05888dfd5cd5d1a78c031 WatchSource:0}: Error finding container cd3f7ce1c69aa6c67c22365c06fa791b404e58f582d05888dfd5cd5d1a78c031: Status 404 returned error can't find the container with id cd3f7ce1c69aa6c67c22365c06fa791b404e58f582d05888dfd5cd5d1a78c031
Sep 17 19:56:42.887907 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.887875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:42.888049 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.887963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8h5\" (UniqueName: \"kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:42.989281 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.989245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:42.989510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.989305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8h5\" (UniqueName: \"kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:42.992664 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:42.992623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:43.000881 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.000807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8h5\" (UniqueName: \"kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:43.062154 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.062114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:43.267997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.267886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerStarted","Data":"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"}
Sep 17 19:56:43.267997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.267958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerStarted","Data":"cd3f7ce1c69aa6c67c22365c06fa791b404e58f582d05888dfd5cd5d1a78c031"}
Sep 17 19:56:43.268513 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.268081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:43.340620 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:43.340581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:56:43.344329 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:56:43.344295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3246a698_7e56_4196_b1a3_cac699534319.slice/crio-c1d0db3704070d43cbb7158e96235f8381942619cda3da4c70adf5f9d611d320 WatchSource:0}: Error finding container c1d0db3704070d43cbb7158e96235f8381942619cda3da4c70adf5f9d611d320: Status 404 returned error can't find the container with id c1d0db3704070d43cbb7158e96235f8381942619cda3da4c70adf5f9d611d320
Sep 17 19:56:44.276081 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:44.276046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerStarted","Data":"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"}
Sep 17 19:56:44.277758 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:44.277731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" event={"ID":"3246a698-7e56-4196-b1a3-cac699534319","Type":"ContainerStarted","Data":"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"}
Sep 17 19:56:44.277892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:44.277784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" event={"ID":"3246a698-7e56-4196-b1a3-cac699534319","Type":"ContainerStarted","Data":"c1d0db3704070d43cbb7158e96235f8381942619cda3da4c70adf5f9d611d320"}
Sep 17 19:56:44.277892 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:44.277812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:56:44.320701 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:44.320633    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podStartSLOduration=2.320616411 podStartE2EDuration="2.320616411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:44.318735939 +0000 UTC m=+3036.332306099" watchObservedRunningTime="2026-09-17 19:56:44.320616411 +0000 UTC m=+3036.334186571"
Sep 17 19:56:46.289938 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:46.289898    2581 generic.go:356] "Generic (PLEG): container finished" podID="677b9a11-9959-40ee-803c-5a062fe1e468" containerID="c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f" exitCode=0
Sep 17 19:56:46.290347 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:46.289982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerDied","Data":"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"}
Sep 17 19:56:47.298742 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:47.298697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerStarted","Data":"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"}
Sep 17 19:56:47.325257 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:47.325194    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podStartSLOduration=3.039240037 podStartE2EDuration="5.325170875s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.285930838s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:47.322667015 +0000 UTC m=+3039.336237176" watchObservedRunningTime="2026-09-17 19:56:47.325170875 +0000 UTC m=+3039.338741041"
Sep 17 19:56:52.544337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:52.544246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:52.544337 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:52.544320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:56:52.545756 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:52.545720    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:56:52.561719 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:56:52.561689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:57:02.544868 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:02.544823    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:57:12.545093 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:12.545033    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:57:15.285224 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:15.285190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:57:22.544973 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:22.544907    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:57:32.545193 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:32.545143    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:57:42.544604 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:42.544555    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:57:52.545045 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:57:52.544990    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:02.544534 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:02.544481    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:12.545119 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:12.545066    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:22.544796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:22.544683    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:32.544919 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:32.544874    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:42.545545 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:42.545491    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8001/health\": dial tcp 10.132.0.65:8001: connect: connection refused"
Sep 17 19:58:52.554609 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:52.554572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:58:52.567457 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:58:52.567402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:59:01.649465 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:01.649409    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:59:01.651471 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:01.651395    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:59:05.285537 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:05.285496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:59:15.284857 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:15.284815    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:59:16.671779 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:16.671741    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" containerID="cri-o://570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b" gracePeriod=45
Sep 17 19:59:16.673762 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:16.673729    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main" containerID="cri-o://97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199" gracePeriod=45
Sep 17 19:59:17.292668 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.292632    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="llm-d-routing-sidecar" containerID="cri-o://c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3" gracePeriod=45
Sep 17 19:59:17.543470 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.543440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:59:17.608146 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608329 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608329 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608329 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml2dj\" (UniqueName: \"kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608510 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm\") pod \"677b9a11-9959-40ee-803c-5a062fe1e468\" (UID: \"677b9a11-9959-40ee-803c-5a062fe1e468\") "
Sep 17 19:59:17.608736 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache" (OuterVolumeSpecName: "model-cache") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.608873 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:59:17.609197 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.608878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home" (OuterVolumeSpecName: "home") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.610865 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.610838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:59:17.610989 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.610958    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj" (OuterVolumeSpecName: "kube-api-access-ml2dj") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "kube-api-access-ml2dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:59:17.611037 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.611013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm" (OuterVolumeSpecName: "dshm") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.622734 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.622668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.633330 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.633292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "677b9a11-9959-40ee-803c-5a062fe1e468" (UID: "677b9a11-9959-40ee-803c-5a062fe1e468"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.709682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709639    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.709682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709673    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677b9a11-9959-40ee-803c-5a062fe1e468-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.709682 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709686    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/677b9a11-9959-40ee-803c-5a062fe1e468-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.710115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709695    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2dj\" (UniqueName: \"kubernetes.io/projected/677b9a11-9959-40ee-803c-5a062fe1e468-kube-api-access-ml2dj\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.710115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709704    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.710115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709713    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.710115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709720    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.710115 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:17.709728    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677b9a11-9959-40ee-803c-5a062fe1e468-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:18.037560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037525    2581 generic.go:356] "Generic (PLEG): container finished" podID="677b9a11-9959-40ee-803c-5a062fe1e468" containerID="97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199" exitCode=0
Sep 17 19:59:18.037560 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037554    2581 generic.go:356] "Generic (PLEG): container finished" podID="677b9a11-9959-40ee-803c-5a062fe1e468" containerID="c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3" exitCode=0
Sep 17 19:59:18.037796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"
Sep 17 19:59:18.037796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerDied","Data":"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"}
Sep 17 19:59:18.037796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerDied","Data":"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"}
Sep 17 19:59:18.037796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2" event={"ID":"677b9a11-9959-40ee-803c-5a062fe1e468","Type":"ContainerDied","Data":"cd3f7ce1c69aa6c67c22365c06fa791b404e58f582d05888dfd5cd5d1a78c031"}
Sep 17 19:59:18.037796 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.037688    2581 scope.go:117] "RemoveContainer" containerID="97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"
Sep 17 19:59:18.047987 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.047964    2581 scope.go:117] "RemoveContainer" containerID="c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"
Sep 17 19:59:18.056697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.056678    2581 scope.go:117] "RemoveContainer" containerID="c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"
Sep 17 19:59:18.062363 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.062337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:59:18.065166 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.065081    2581 scope.go:117] "RemoveContainer" containerID="97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"
Sep 17 19:59:18.065565 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:59:18.065540    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199\": container with ID starting with 97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199 not found: ID does not exist" containerID="97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"
Sep 17 19:59:18.065645 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.065581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"} err="failed to get container status \"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199\": rpc error: code = NotFound desc = could not find container \"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199\": container with ID starting with 97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199 not found: ID does not exist"
Sep 17 19:59:18.065645 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.065605    2581 scope.go:117] "RemoveContainer" containerID="c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"
Sep 17 19:59:18.065934 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:59:18.065914    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f\": container with ID starting with c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f not found: ID does not exist" containerID="c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"
Sep 17 19:59:18.065997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.065940    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"} err="failed to get container status \"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f\": rpc error: code = NotFound desc = could not find container \"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f\": container with ID starting with c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f not found: ID does not exist"
Sep 17 19:59:18.065997 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.065962    2581 scope.go:117] "RemoveContainer" containerID="c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"
Sep 17 19:59:18.066227 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:59:18.066212    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3\": container with ID starting with c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3 not found: ID does not exist" containerID="c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"
Sep 17 19:59:18.066276 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066230    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"} err="failed to get container status \"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3\": rpc error: code = NotFound desc = could not find container \"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3\": container with ID starting with c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3 not found: ID does not exist"
Sep 17 19:59:18.066276 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066244    2581 scope.go:117] "RemoveContainer" containerID="97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"
Sep 17 19:59:18.066558 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066533    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199"} err="failed to get container status \"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199\": rpc error: code = NotFound desc = could not find container \"97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199\": container with ID starting with 97181d247fd52425642fad83342c0d8727febaf6c54fe28c65c17bf898ba9199 not found: ID does not exist"
Sep 17 19:59:18.066637 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066559    2581 scope.go:117] "RemoveContainer" containerID="c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"
Sep 17 19:59:18.066803 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f"} err="failed to get container status \"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f\": rpc error: code = NotFound desc = could not find container \"c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f\": container with ID starting with c0774155952d63fb17acf3efaa006b3f7322494fdd19271f37765d9dc41c977f not found: ID does not exist"
Sep 17 19:59:18.066846 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066804    2581 scope.go:117] "RemoveContainer" containerID="c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"
Sep 17 19:59:18.066846 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.066794    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-599dcc896b-h48m2"]
Sep 17 19:59:18.067060 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.067042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3"} err="failed to get container status \"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3\": rpc error: code = NotFound desc = could not find container \"c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3\": container with ID starting with c62e5d4c7fbb501728b487d778623816415eb3c4d9e857cc35cd589692f4d7b3 not found: ID does not exist"
Sep 17 19:59:18.137816 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:18.137781    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" path="/var/lib/kubelet/pods/677b9a11-9959-40ee-803c-5a062fe1e468/volumes"
Sep 17 19:59:25.285765 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:25.285729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:59:25.286198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:25.285835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:59:35.285505 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:35.285438    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:59:45.285706 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:45.285663    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:59:46.934326 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:46.934257    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:59:47.071104 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.071062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b8h5\" (UniqueName: \"kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5\") pod \"3246a698-7e56-4196-b1a3-cac699534319\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") "
Sep 17 19:59:47.071282 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.071125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs\") pod \"3246a698-7e56-4196-b1a3-cac699534319\" (UID: \"3246a698-7e56-4196-b1a3-cac699534319\") "
Sep 17 19:59:47.073511 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.073474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5" (OuterVolumeSpecName: "kube-api-access-2b8h5") pod "3246a698-7e56-4196-b1a3-cac699534319" (UID: "3246a698-7e56-4196-b1a3-cac699534319"). InnerVolumeSpecName "kube-api-access-2b8h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:59:47.073652 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.073520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3246a698-7e56-4196-b1a3-cac699534319" (UID: "3246a698-7e56-4196-b1a3-cac699534319"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:59:47.172189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.172146    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8h5\" (UniqueName: \"kubernetes.io/projected/3246a698-7e56-4196-b1a3-cac699534319-kube-api-access-2b8h5\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:47.172189 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.172181    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246a698-7e56-4196-b1a3-cac699534319-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 19:59:47.182532 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.182501    2581 generic.go:356] "Generic (PLEG): container finished" podID="3246a698-7e56-4196-b1a3-cac699534319" containerID="570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b" exitCode=0
Sep 17 19:59:47.182695 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.182544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"
Sep 17 19:59:47.182695 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.182560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" event={"ID":"3246a698-7e56-4196-b1a3-cac699534319","Type":"ContainerDied","Data":"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"}
Sep 17 19:59:47.182695 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.182607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4" event={"ID":"3246a698-7e56-4196-b1a3-cac699534319","Type":"ContainerDied","Data":"c1d0db3704070d43cbb7158e96235f8381942619cda3da4c70adf5f9d611d320"}
Sep 17 19:59:47.182695 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.182636    2581 scope.go:117] "RemoveContainer" containerID="570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"
Sep 17 19:59:47.195591 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.195570    2581 scope.go:117] "RemoveContainer" containerID="570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"
Sep 17 19:59:47.195907 ip-10-0-132-57 kubenswrapper[2581]: E0917 19:59:47.195888    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b\": container with ID starting with 570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b not found: ID does not exist" containerID="570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"
Sep 17 19:59:47.195964 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.195916    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b"} err="failed to get container status \"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b\": rpc error: code = NotFound desc = could not find container \"570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b\": container with ID starting with 570aae8a9b878c596dfd16d64a1a61a7bbea69781562c9078c21b6b047b97f2b not found: ID does not exist"
Sep 17 19:59:47.207535 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.207508    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:59:47.211223 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:47.211194    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45znld4"]
Sep 17 19:59:48.137778 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:48.137744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3246a698-7e56-4196-b1a3-cac699534319" path="/var/lib/kubelet/pods/3246a698-7e56-4196-b1a3-cac699534319/volumes"
Sep 17 19:59:56.684731 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:56.684642    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 19:59:56.685136 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:56.684886    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy" containerID="cri-o://6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c" gracePeriod=30
Sep 17 19:59:57.664058 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664016    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 19:59:57.664398 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664382    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664400    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="llm-d-routing-sidecar"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664434    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="llm-d-routing-sidecar"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664448    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="storage-initializer"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="storage-initializer"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664459    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main"
Sep 17 19:59:57.664481 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main"
Sep 17 19:59:57.664697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664514    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="llm-d-routing-sidecar"
Sep 17 19:59:57.664697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664520    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3246a698-7e56-4196-b1a3-cac699534319" containerName="main"
Sep 17 19:59:57.664697 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.664528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="677b9a11-9959-40ee-803c-5a062fe1e468" containerName="main"
Sep 17 19:59:57.668934 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.668915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.671470 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.671440    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:59:57.671614 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.671479    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 19:59:57.671614 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.671530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 19:59:57.671614 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.671532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-j7rmp\""
Sep 17 19:59:57.672208 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.672188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 19:59:57.672324 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.672215    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 19:59:57.682362 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.682323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 19:59:57.763780 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffpj\" (UniqueName: \"kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.764254 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.763999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.864851 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.864802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.864851 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.864850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.864875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.864986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865127 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffpj\" (UniqueName: \"kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865395 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865576 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.865942 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.865923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.867455 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.867432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.867685 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.867666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.873198 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.873179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffpj\" (UniqueName: \"kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:57.980446 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:57.980377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 19:59:58.267401 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:58.267276    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 19:59:58.273031 ip-10-0-132-57 kubenswrapper[2581]: W0917 19:59:58.272996    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e568d7f_cfc4_4a02_8b25_3f4ac79ca315.slice/crio-c4647def99b64532618b102513095e4254433da6143392d7999b754efd77abdc WatchSource:0}: Error finding container c4647def99b64532618b102513095e4254433da6143392d7999b754efd77abdc: Status 404 returned error can't find the container with id c4647def99b64532618b102513095e4254433da6143392d7999b754efd77abdc
Sep 17 19:59:59.247980 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:59.247942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerStarted","Data":"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"}
Sep 17 19:59:59.248373 ip-10-0-132-57 kubenswrapper[2581]: I0917 19:59:59.248038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerStarted","Data":"c4647def99b64532618b102513095e4254433da6143392d7999b754efd77abdc"}
Sep 17 20:00:01.260181 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:01.260143    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerID="277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5" exitCode=0
Sep 17 20:00:01.260619 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:01.260187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerDied","Data":"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"}
Sep 17 20:00:01.941958 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:01.941928    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 20:00:02.100551 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100551 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100820 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100820 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100820 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czhb4\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs\") pod \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\" (UID: \"e9e33a32-a294-447e-bea0-2b2ce6e884ba\") "
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100932    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:02.100989 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.100944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101269    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-credential-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101288    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-socket\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101301    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-data\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.101291 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.101314    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istiod-ca-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.103115 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.103091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 20:00:02.103115 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.103107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:02.103584 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.103564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:00:02.103666 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.103587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4" (OuterVolumeSpecName: "kube-api-access-czhb4") pod "e9e33a32-a294-447e-bea0-2b2ce6e884ba" (UID: "e9e33a32-a294-447e-bea0-2b2ce6e884ba"). InnerVolumeSpecName "kube-api-access-czhb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:00:02.202277 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.202237    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-podinfo\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.202277 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.202268    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-token\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.202277 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.202277    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhb4\" (UniqueName: \"kubernetes.io/projected/e9e33a32-a294-447e-bea0-2b2ce6e884ba-kube-api-access-czhb4\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.202277 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.202286    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-workload-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.202583 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.202295    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e33a32-a294-447e-bea0-2b2ce6e884ba-istio-envoy\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:00:02.266516 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.266433    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerID="6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c" exitCode=0
Sep 17 20:00:02.266516 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.266489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"
Sep 17 20:00:02.267013 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.266527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerDied","Data":"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"}
Sep 17 20:00:02.267013 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.266598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k" event={"ID":"e9e33a32-a294-447e-bea0-2b2ce6e884ba","Type":"ContainerDied","Data":"e8500269e265cb22248a94ae1c375ab3a5de665b4f1c4ebd6469ef1bb32a7b7d"}
Sep 17 20:00:02.267013 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.266635    2581 scope.go:117] "RemoveContainer" containerID="6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"
Sep 17 20:00:02.268967 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.268939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerStarted","Data":"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"}
Sep 17 20:00:02.276876 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.276850    2581 scope.go:117] "RemoveContainer" containerID="828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"
Sep 17 20:00:02.284939 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.284908    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 20:00:02.285797 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.285753    2581 scope.go:117] "RemoveContainer" containerID="6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"
Sep 17 20:00:02.286350 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:00:02.286328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c\": container with ID starting with 6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c not found: ID does not exist" containerID="6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"
Sep 17 20:00:02.286440 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.286359    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c"} err="failed to get container status \"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c\": rpc error: code = NotFound desc = could not find container \"6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c\": container with ID starting with 6ca417243d16d9cd17379db7518e03a1d5cd98a16993942108d18f80e18ed35c not found: ID does not exist"
Sep 17 20:00:02.286440 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.286377    2581 scope.go:117] "RemoveContainer" containerID="828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"
Sep 17 20:00:02.286721 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:00:02.286703    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897\": container with ID starting with 828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897 not found: ID does not exist" containerID="828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"
Sep 17 20:00:02.286780 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.286725    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897"} err="failed to get container status \"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897\": rpc error: code = NotFound desc = could not find container \"828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897\": container with ID starting with 828ff29f8839533eb4edd1560d6102f6e718c6f0c18f6269584b0c22a1543897 not found: ID does not exist"
Sep 17 20:00:02.287691 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.287673    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-57b79bd49b-qwb6k"]
Sep 17 20:00:02.306395 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.306348    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podStartSLOduration=3.00393097 podStartE2EDuration="5.306328776s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.302397806s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:00:02.305454859 +0000 UTC m=+3234.319025020" watchObservedRunningTime="2026-09-17 20:00:02.306328776 +0000 UTC m=+3234.319898937"
Sep 17 20:00:02.981383 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:02.981347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:00:04.137138 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:04.137100    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" path="/var/lib/kubelet/pods/e9e33a32-a294-447e-bea0-2b2ce6e884ba/volumes"
Sep 17 20:00:07.980820 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:07.980773    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:00:07.982295 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:07.982256    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:00:17.981899 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:17.981842    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:00:27.981917 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:27.981867    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:00:37.980849 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:37.980803    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:00:47.981819 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:47.981765    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:00:57.981346 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:00:57.981227    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:06.175940 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.175903    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:01:06.176455 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.176250    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:01:06.176455 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.176263    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:01:06.176455 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.176333    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:01:06.179559 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.179529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.183013 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.182989    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 20:01:06.183189 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.183161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 20:01:06.183326 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.183079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 20:01:06.183395 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.183085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-rkswm\""
Sep 17 20:01:06.183539 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.183080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-x2xpm\""
Sep 17 20:01:06.190630 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.190604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:01:06.272244 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.272202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kbn\" (UniqueName: \"kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.272487 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.272270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.373022 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.372983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kbn\" (UniqueName: \"kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.373233 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.373032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.375742 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.375716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.381569 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.381545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kbn\" (UniqueName: \"kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.494597 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.494493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:06.771514 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:06.771467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:01:06.775008 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:01:06.774956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode561fa51_0595_4a78_9732_04b44f431a56.slice/crio-ea715b861fe4fa53c7a69ee27ca59b2b37a54dc377fbfa5751c18e05f901ad14 WatchSource:0}: Error finding container ea715b861fe4fa53c7a69ee27ca59b2b37a54dc377fbfa5751c18e05f901ad14: Status 404 returned error can't find the container with id ea715b861fe4fa53c7a69ee27ca59b2b37a54dc377fbfa5751c18e05f901ad14
Sep 17 20:01:07.604983 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:07.604936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" event={"ID":"e561fa51-0595-4a78-9732-04b44f431a56","Type":"ContainerStarted","Data":"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"}
Sep 17 20:01:07.605444 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:07.605007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" event={"ID":"e561fa51-0595-4a78-9732-04b44f431a56","Type":"ContainerStarted","Data":"ea715b861fe4fa53c7a69ee27ca59b2b37a54dc377fbfa5751c18e05f901ad14"}
Sep 17 20:01:07.605444 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:07.605055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:07.623623 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:07.623553    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podStartSLOduration=1.62352733 podStartE2EDuration="1.62352733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:01:07.621306343 +0000 UTC m=+3299.634876504" watchObservedRunningTime="2026-09-17 20:01:07.62352733 +0000 UTC m=+3299.637097491"
Sep 17 20:01:07.981000 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:07.980959    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:10.363002 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:10.362971    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 20:01:10.373209 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:10.373179    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 20:01:17.981232 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:17.981180    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:27.981715 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:27.981612    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:37.981183 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:37.981140    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:38.612475 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:38.612437    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:01:47.981145 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:47.981099    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:01:57.981787 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:01:57.981741    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:02:07.991081 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:07.991038    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:02:07.998972 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:07.998945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:02:12.662013 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:12.661973    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 20:02:27.685070 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:27.685001    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" containerID="cri-o://55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe" gracePeriod=45
Sep 17 20:02:27.996841 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:27.996723    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 20:02:28.444029 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.443996    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:02:28.494566 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494566 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zffpj\" (UniqueName: \"kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.494835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs\") pod \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\" (UID: \"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315\") "
Sep 17 20:02:28.495156 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.494962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:02:28.495216 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.495186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:02:28.495464 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.495402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home" (OuterVolumeSpecName: "home") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:02:28.497279 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.497246    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm" (OuterVolumeSpecName: "dshm") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:02:28.497369 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.497323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:02:28.497615 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.497449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj" (OuterVolumeSpecName: "kube-api-access-zffpj") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "kube-api-access-zffpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:02:28.513027 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.512987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:02:28.522763 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.522725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" (UID: "6e568d7f-cfc4-4a02-8b25-3f4ac79ca315"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596321    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596360    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596371    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596381    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zffpj\" (UniqueName: \"kubernetes.io/projected/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-kube-api-access-zffpj\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596389    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596398    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596407    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.596446 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.596441    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:02:28.983864 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.983825    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerID="55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe" exitCode=0
Sep 17 20:02:28.984358 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.983880    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"
Sep 17 20:02:28.984358 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.983914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerDied","Data":"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"}
Sep 17 20:02:28.984358 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.983984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv" event={"ID":"6e568d7f-cfc4-4a02-8b25-3f4ac79ca315","Type":"ContainerDied","Data":"c4647def99b64532618b102513095e4254433da6143392d7999b754efd77abdc"}
Sep 17 20:02:28.984358 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.984010    2581 scope.go:117] "RemoveContainer" containerID="55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"
Sep 17 20:02:28.993849 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:28.993827    2581 scope.go:117] "RemoveContainer" containerID="277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"
Sep 17 20:02:29.001984 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.001962    2581 scope.go:117] "RemoveContainer" containerID="55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"
Sep 17 20:02:29.002291 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:02:29.002271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe\": container with ID starting with 55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe not found: ID does not exist" containerID="55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"
Sep 17 20:02:29.002337 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.002300    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe"} err="failed to get container status \"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe\": rpc error: code = NotFound desc = could not find container \"55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe\": container with ID starting with 55767a6c74e0d67091dd2626a4805ebb1042e856b22c35d685b886ec340eebbe not found: ID does not exist"
Sep 17 20:02:29.002337 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.002320    2581 scope.go:117] "RemoveContainer" containerID="277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"
Sep 17 20:02:29.002572 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:02:29.002549    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5\": container with ID starting with 277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5 not found: ID does not exist" containerID="277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"
Sep 17 20:02:29.002619 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.002581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5"} err="failed to get container status \"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5\": rpc error: code = NotFound desc = could not find container \"277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5\": container with ID starting with 277a3235096d7d732894c027cdac209087f8b2ba0c78da64dbc2e74b61a5b2b5 not found: ID does not exist"
Sep 17 20:02:29.007292 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.007269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 20:02:29.013731 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:29.013706    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-58f5cd649-x9nvv"]
Sep 17 20:02:30.137666 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:30.137633    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" path="/var/lib/kubelet/pods/6e568d7f-cfc4-4a02-8b25-3f4ac79ca315/volumes"
Sep 17 20:02:50.280311 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280212    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="storage-initializer"
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280718    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="storage-initializer"
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280740    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main"
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280748    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main"
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:02:50.280827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280771    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:02:50.281064 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280848    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e568d7f-cfc4-4a02-8b25-3f4ac79ca315" containerName="main"
Sep 17 20:02:50.281064 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.280861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e33a32-a294-447e-bea0-2b2ce6e884ba" containerName="istio-proxy"
Sep 17 20:02:50.283998 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.283976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.287426 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.287390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 20:02:50.287956 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.287937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 20:02:50.288087 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.287969    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 20:02:50.288087 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.287942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-dzcdg\""
Sep 17 20:02:50.288087 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.288015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-9f8gb\""
Sep 17 20:02:50.296457 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.296399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:02:50.365854 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.365817    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:02:50.369156 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.369138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.374961 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.374933    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 20:02:50.375102 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.375000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 20:02:50.376161 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.375984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rhnr7\""
Sep 17 20:02:50.381308 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.381288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.381398 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.381331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjgd\" (UniqueName: \"kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.391262 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.391241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:02:50.481907 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.481861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.481907 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.481913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482200 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.481966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjgd\" (UniqueName: \"kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.482200 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztqd\" (UniqueName: \"kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482200 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482200 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482200 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482530 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.482530 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.482530 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.482346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.485202 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.485174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.490612 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.490588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjgd\" (UniqueName: \"kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.583092 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.582975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583092 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583092 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583092 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztqd\" (UniqueName: \"kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583092 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583541 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583541 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583541 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583541 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583541 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583868 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.583942 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.583888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.584141 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.584114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.585752 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.585728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.585997 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.585977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.593693 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.593671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztqd\" (UniqueName: \"kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.596545 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.596523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:50.679836 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.679806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:02:50.873830 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.873803    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:02:50.875968 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:02:50.875934    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9737631e_ad51_4e8d_8caa_97cf143ad7f8.slice/crio-60b6292be4984691a1fb2ddf5f02b2cd8495f307c63459295cd2c21ce34970e0 WatchSource:0}: Error finding container 60b6292be4984691a1fb2ddf5f02b2cd8495f307c63459295cd2c21ce34970e0: Status 404 returned error can't find the container with id 60b6292be4984691a1fb2ddf5f02b2cd8495f307c63459295cd2c21ce34970e0
Sep 17 20:02:50.968461 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:50.968370    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:02:50.972153 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:02:50.972114    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a325c5_8681_480b_a3d8_fd9fb0fc10c3.slice/crio-bd5b1097b1709e997e0893a886676eb66186437f9fc93e39892377f774041829 WatchSource:0}: Error finding container bd5b1097b1709e997e0893a886676eb66186437f9fc93e39892377f774041829: Status 404 returned error can't find the container with id bd5b1097b1709e997e0893a886676eb66186437f9fc93e39892377f774041829
Sep 17 20:02:51.088801 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.088753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" event={"ID":"9737631e-ad51-4e8d-8caa-97cf143ad7f8","Type":"ContainerStarted","Data":"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"}
Sep 17 20:02:51.089017 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.088876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" event={"ID":"9737631e-ad51-4e8d-8caa-97cf143ad7f8","Type":"ContainerStarted","Data":"60b6292be4984691a1fb2ddf5f02b2cd8495f307c63459295cd2c21ce34970e0"}
Sep 17 20:02:51.089017 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.088911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:02:51.091201 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.091166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerStarted","Data":"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"}
Sep 17 20:02:51.091330 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.091207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerStarted","Data":"bd5b1097b1709e997e0893a886676eb66186437f9fc93e39892377f774041829"}
Sep 17 20:02:51.107877 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:51.107808    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podStartSLOduration=1.1077864 podStartE2EDuration="1.1077864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:02:51.106159114 +0000 UTC m=+3403.119729276" watchObservedRunningTime="2026-09-17 20:02:51.1077864 +0000 UTC m=+3403.121356562"
Sep 17 20:02:54.108270 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:54.108234    2581 generic.go:356] "Generic (PLEG): container finished" podID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerID="a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087" exitCode=0
Sep 17 20:02:54.108774 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:54.108311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerDied","Data":"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"}
Sep 17 20:02:55.114590 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:55.114554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerStarted","Data":"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"}
Sep 17 20:02:55.136208 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:02:55.136151    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.7911235420000002 podStartE2EDuration="5.136128324s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.345004782s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:02:55.134763965 +0000 UTC m=+3407.148334160" watchObservedRunningTime="2026-09-17 20:02:55.136128324 +0000 UTC m=+3407.149698485"
Sep 17 20:03:22.098137 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:22.098102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:03:27.720464 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:27.720404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:03:28.612102 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:28.612060    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:03:38.611828 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:38.611784    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:03:42.746035 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:42.745987    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" containerID="cri-o://0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8" gracePeriod=45
Sep 17 20:03:48.611752 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:48.611704    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:03:48.612266 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:48.611829    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:03:58.612389 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:03:58.612344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:04:08.612569 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:08.612531    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:04:12.986835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:12.986797    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:04:13.112314 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.112203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2kbn\" (UniqueName: \"kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn\") pod \"e561fa51-0595-4a78-9732-04b44f431a56\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") "
Sep 17 20:04:13.112314 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.112267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs\") pod \"e561fa51-0595-4a78-9732-04b44f431a56\" (UID: \"e561fa51-0595-4a78-9732-04b44f431a56\") "
Sep 17 20:04:13.114776 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.114745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn" (OuterVolumeSpecName: "kube-api-access-h2kbn") pod "e561fa51-0595-4a78-9732-04b44f431a56" (UID: "e561fa51-0595-4a78-9732-04b44f431a56"). InnerVolumeSpecName "kube-api-access-h2kbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:04:13.114929 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.114808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e561fa51-0595-4a78-9732-04b44f431a56" (UID: "e561fa51-0595-4a78-9732-04b44f431a56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:04:13.212932 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.212885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kbn\" (UniqueName: \"kubernetes.io/projected/e561fa51-0595-4a78-9732-04b44f431a56-kube-api-access-h2kbn\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:04:13.212932 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.212923    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e561fa51-0595-4a78-9732-04b44f431a56-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:04:13.495632 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.495593    2581 generic.go:356] "Generic (PLEG): container finished" podID="e561fa51-0595-4a78-9732-04b44f431a56" containerID="0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8" exitCode=0
Sep 17 20:04:13.495851 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.495638    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"
Sep 17 20:04:13.495851 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.495675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" event={"ID":"e561fa51-0595-4a78-9732-04b44f431a56","Type":"ContainerDied","Data":"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"}
Sep 17 20:04:13.495851 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.495731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz" event={"ID":"e561fa51-0595-4a78-9732-04b44f431a56","Type":"ContainerDied","Data":"ea715b861fe4fa53c7a69ee27ca59b2b37a54dc377fbfa5751c18e05f901ad14"}
Sep 17 20:04:13.495851 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.495758    2581 scope.go:117] "RemoveContainer" containerID="0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"
Sep 17 20:04:13.506583 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.506564    2581 scope.go:117] "RemoveContainer" containerID="0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"
Sep 17 20:04:13.506883 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:04:13.506859    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8\": container with ID starting with 0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8 not found: ID does not exist" containerID="0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"
Sep 17 20:04:13.506960 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.506892    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8"} err="failed to get container status \"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8\": rpc error: code = NotFound desc = could not find container \"0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8\": container with ID starting with 0744d560d0dfaab0121ce9ba0fed643012893269e30495c12df2d992f86a9bc8 not found: ID does not exist"
Sep 17 20:04:13.519022 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.518992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:04:13.523991 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:13.523963    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-445vz"]
Sep 17 20:04:14.138019 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:04:14.137982    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e561fa51-0595-4a78-9732-04b44f431a56" path="/var/lib/kubelet/pods/e561fa51-0595-4a78-9732-04b44f431a56/volumes"
Sep 17 20:05:06.381400 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:06.381363    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:05:12.097872 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:12.097831    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:05:14.849807 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.849767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:14.850195 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.850079    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main"
Sep 17 20:05:14.850195 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.850090    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main"
Sep 17 20:05:14.850195 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.850152    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e561fa51-0595-4a78-9732-04b44f431a56" containerName="main"
Sep 17 20:05:14.852969 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.852951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.856257 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.856229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-gll76\""
Sep 17 20:05:14.856395 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.856268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 20:05:14.856395 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.856320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 20:05:14.856395 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.856348    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 20:05:14.865221 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.865197    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:14.945768 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.945957 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.945957 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74dv\" (UniqueName: \"kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.945957 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.945957 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:14.945957 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:14.945928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046532 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74dv\" (UniqueName: \"kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.046966 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.047022 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.046994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.047075 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.047057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.049167 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.049142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.049493 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.049475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.054523 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.054501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74dv\" (UniqueName: \"kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-slnnx\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.163232 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.163142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:15.436154 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.436125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:15.438996 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:05:15.438960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3514cd03_cc6f_4645_b38d_9d3b2f8ad9ff.slice/crio-a895ef53e4d12bdeda40ad79be0c1cd4dbe465391013518fb8f6df7ebed0419e WatchSource:0}: Error finding container a895ef53e4d12bdeda40ad79be0c1cd4dbe465391013518fb8f6df7ebed0419e: Status 404 returned error can't find the container with id a895ef53e4d12bdeda40ad79be0c1cd4dbe465391013518fb8f6df7ebed0419e
Sep 17 20:05:15.800455 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.800343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" event={"ID":"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff","Type":"ContainerStarted","Data":"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"}
Sep 17 20:05:15.800455 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.800392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" event={"ID":"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff","Type":"ContainerStarted","Data":"a895ef53e4d12bdeda40ad79be0c1cd4dbe465391013518fb8f6df7ebed0419e"}
Sep 17 20:05:15.818390 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:15.818318    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" podStartSLOduration=1.818304469 podStartE2EDuration="1.818304469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:05:15.817233972 +0000 UTC m=+3547.830804133" watchObservedRunningTime="2026-09-17 20:05:15.818304469 +0000 UTC m=+3547.831874630"
Sep 17 20:05:16.164379 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:16.164278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:21.403203 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:21.403156    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main" containerID="cri-o://67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39" gracePeriod=45
Sep 17 20:05:22.097644 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:22.097603    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:05:22.997345 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:22.997304    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:05:25.163776 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:25.163730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:25.176149 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:25.176120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:25.857792 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:25.857759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:32.097742 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:32.097698    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:05:32.098148 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:32.097815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:05:38.020316 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.020270    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="main" containerID="cri-o://4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f" gracePeriod=45
Sep 17 20:05:38.770256 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.770228    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:05:38.837060 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.836980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837060 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ztqd\" (UniqueName: \"kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837282 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert\") pod \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\" (UID: \"55a325c5-8681-480b-a3d8-fd9fb0fc10c3\") "
Sep 17 20:05:38.837895 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.837837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:38.838098 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.838056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:05:38.838184 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.838151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home" (OuterVolumeSpecName: "home") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:38.839731 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.839690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:05:38.839835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.839775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm" (OuterVolumeSpecName: "dshm") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:38.839835 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.839784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd" (OuterVolumeSpecName: "kube-api-access-6ztqd") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "kube-api-access-6ztqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:05:38.855009 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.854978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:38.868483 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.868452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55a325c5-8681-480b-a3d8-fd9fb0fc10c3" (UID: "55a325c5-8681-480b-a3d8-fd9fb0fc10c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:38.912679 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.912639    2581 generic.go:356] "Generic (PLEG): container finished" podID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerID="4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f" exitCode=0
Sep 17 20:05:38.912816 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.912689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 20:05:38.912816 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.912733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerDied","Data":"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"}
Sep 17 20:05:38.912816 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.912789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"55a325c5-8681-480b-a3d8-fd9fb0fc10c3","Type":"ContainerDied","Data":"bd5b1097b1709e997e0893a886676eb66186437f9fc93e39892377f774041829"}
Sep 17 20:05:38.912816 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.912816    2581 scope.go:117] "RemoveContainer" containerID="4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"
Sep 17 20:05:38.922730 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.922711    2581 scope.go:117] "RemoveContainer" containerID="a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"
Sep 17 20:05:38.930878 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.930862    2581 scope.go:117] "RemoveContainer" containerID="4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"
Sep 17 20:05:38.931175 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:38.931156    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f\": container with ID starting with 4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f not found: ID does not exist" containerID="4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"
Sep 17 20:05:38.931216 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.931185    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f"} err="failed to get container status \"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f\": rpc error: code = NotFound desc = could not find container \"4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f\": container with ID starting with 4fdb0330fdb28d9f6ef00179688c4cdc983b85acfc10982db885182b2c78411f not found: ID does not exist"
Sep 17 20:05:38.931216 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.931202    2581 scope.go:117] "RemoveContainer" containerID="a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"
Sep 17 20:05:38.931482 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:38.931461    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087\": container with ID starting with a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087 not found: ID does not exist" containerID="a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"
Sep 17 20:05:38.931546 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.931487    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087"} err="failed to get container status \"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087\": rpc error: code = NotFound desc = could not find container \"a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087\": container with ID starting with a4b72fcd3ffc87480f3d3f8a76881117e5f1056c12731ac048a586d53832f087 not found: ID does not exist"
Sep 17 20:05:38.937016 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.936992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:05:38.938561 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938543    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938564    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938572    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938581    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztqd\" (UniqueName: \"kubernetes.io/projected/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kube-api-access-6ztqd\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938590    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938598    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938607    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-kserve-provision-location\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.938614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.938614    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55a325c5-8681-480b-a3d8-fd9fb0fc10c3-cabundle-cert\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:38.941304 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:38.941282    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 20:05:40.136759 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:40.136728    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" path="/var/lib/kubelet/pods/55a325c5-8681-480b-a3d8-fd9fb0fc10c3/volumes"
Sep 17 20:05:41.882394 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:41.882344    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:41.893140 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:41.893085    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 20:05:41.893140 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:41.893140 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:41.893140 ip-10-0-132-57 kubenswrapper[2581]:  > containerID="0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc" cmd=["/bin/sleep","15"]
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:41.893150    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" message="" err=<
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:  >
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:41.893171    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerName="main" containerID="cri-o://0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"
Sep 17 20:05:41.893521 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:41.893217    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerName="main" containerID="cri-o://0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc" gracePeriod=60
Sep 17 20:05:42.098587 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.098547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:05:42.251898 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.251872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:42.373372 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373562 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373461    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373562 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373562 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373562 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p74dv\" (UniqueName: \"kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373562 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm\") pod \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\" (UID: \"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff\") "
Sep 17 20:05:42.373830 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:42.373830 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home" (OuterVolumeSpecName: "home") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:42.373830 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:42.373977 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.373843    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-model-cache\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.376018 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.375986    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:05:42.376122 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.376020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm" (OuterVolumeSpecName: "dshm") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:42.376122 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.376036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv" (OuterVolumeSpecName: "kube-api-access-p74dv") pod "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" (UID: "3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff"). InnerVolumeSpecName "kube-api-access-p74dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:05:42.475026 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.474987    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-dshm\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.475026 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.475022    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-home\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.475026 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.475032    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.475269 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.475042    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-tmp-dir\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.475269 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.475058    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74dv\" (UniqueName: \"kubernetes.io/projected/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff-kube-api-access-p74dv\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:42.932530 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.932494    2581 generic.go:356] "Generic (PLEG): container finished" podID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerID="0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc" exitCode=0
Sep 17 20:05:42.933007 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.932544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"
Sep 17 20:05:42.933007 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.932581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" event={"ID":"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff","Type":"ContainerDied","Data":"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"}
Sep 17 20:05:42.933007 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.932639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx" event={"ID":"3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff","Type":"ContainerDied","Data":"a895ef53e4d12bdeda40ad79be0c1cd4dbe465391013518fb8f6df7ebed0419e"}
Sep 17 20:05:42.933007 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.932661    2581 scope.go:117] "RemoveContainer" containerID="0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"
Sep 17 20:05:42.941940 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.941922    2581 scope.go:117] "RemoveContainer" containerID="0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"
Sep 17 20:05:42.942213 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:42.942197    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc\": container with ID starting with 0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc not found: ID does not exist" containerID="0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"
Sep 17 20:05:42.942263 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.942222    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc"} err="failed to get container status \"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc\": rpc error: code = NotFound desc = could not find container \"0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc\": container with ID starting with 0f5e9792936a5a9792a1b56df521654d53ac24260b58cfd5173ac250e32498dc not found: ID does not exist"
Sep 17 20:05:42.954309 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.954277    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:42.959678 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:42.959656    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-slnnx"]
Sep 17 20:05:44.137361 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:44.137318    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" path="/var/lib/kubelet/pods/3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff/volumes"
Sep 17 20:05:51.662998 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.662930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:05:51.753744 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.753709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrjgd\" (UniqueName: \"kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd\") pod \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") "
Sep 17 20:05:51.753744 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.753752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs\") pod \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\" (UID: \"9737631e-ad51-4e8d-8caa-97cf143ad7f8\") "
Sep 17 20:05:51.756163 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.756129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9737631e-ad51-4e8d-8caa-97cf143ad7f8" (UID: "9737631e-ad51-4e8d-8caa-97cf143ad7f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:05:51.756262 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.756180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd" (OuterVolumeSpecName: "kube-api-access-mrjgd") pod "9737631e-ad51-4e8d-8caa-97cf143ad7f8" (UID: "9737631e-ad51-4e8d-8caa-97cf143ad7f8"). InnerVolumeSpecName "kube-api-access-mrjgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:05:51.854595 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.854555    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjgd\" (UniqueName: \"kubernetes.io/projected/9737631e-ad51-4e8d-8caa-97cf143ad7f8-kube-api-access-mrjgd\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:51.854595 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.854588    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9737631e-ad51-4e8d-8caa-97cf143ad7f8-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:05:51.972706 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.972671    2581 generic.go:356] "Generic (PLEG): container finished" podID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerID="67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39" exitCode=0
Sep 17 20:05:51.972706 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.972711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" event={"ID":"9737631e-ad51-4e8d-8caa-97cf143ad7f8","Type":"ContainerDied","Data":"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"}
Sep 17 20:05:51.972935 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.972722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"
Sep 17 20:05:51.972935 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.972752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6" event={"ID":"9737631e-ad51-4e8d-8caa-97cf143ad7f8","Type":"ContainerDied","Data":"60b6292be4984691a1fb2ddf5f02b2cd8495f307c63459295cd2c21ce34970e0"}
Sep 17 20:05:51.972935 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.972774    2581 scope.go:117] "RemoveContainer" containerID="67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"
Sep 17 20:05:51.981910 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.981884    2581 scope.go:117] "RemoveContainer" containerID="67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"
Sep 17 20:05:51.982207 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:05:51.982183    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39\": container with ID starting with 67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39 not found: ID does not exist" containerID="67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"
Sep 17 20:05:51.982259 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.982218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39"} err="failed to get container status \"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39\": rpc error: code = NotFound desc = could not find container \"67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39\": container with ID starting with 67e26862017e6aea32aba3044231163fd8a34fe6c84025dfcb186440ea6afd39 not found: ID does not exist"
Sep 17 20:05:51.994748 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.994718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:05:51.998972 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:51.998945    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6llw6"]
Sep 17 20:05:52.137364 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:52.137321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" path="/var/lib/kubelet/pods/9737631e-ad51-4e8d-8caa-97cf143ad7f8/volumes"
Sep 17 20:05:59.097822 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.097785    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098108    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098125    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="storage-initializer"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098131    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="storage-initializer"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098160    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098165    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098214    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a325c5-8681-480b-a3d8-fd9fb0fc10c3" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098222    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3514cd03-cc6f-4645-b38d-9d3b2f8ad9ff" containerName="main"
Sep 17 20:05:59.100298 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.098229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9737631e-ad51-4e8d-8caa-97cf143ad7f8" containerName="main"
Sep 17 20:05:59.101254 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.101234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.104677 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.104651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-8x6vg\""
Sep 17 20:05:59.104921 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.104696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 20:05:59.105057 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.104755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 20:05:59.105057 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.104777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 20:05:59.105057 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.104826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-f6k74\""
Sep 17 20:05:59.111919 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.111896    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:05:59.219311 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.219271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.219513 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.219316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.319861 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.319820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.319861 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.319864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.322707 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.322679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.329052 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.329027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.412269 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.412183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:05:59.697047 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:05:59.696965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:05:59.699593 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:05:59.699555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab23dd_cc9d_41e4_9682_9df5e5584230.slice/crio-fe1b4b813c7c6f3a1c289c5f01c66784889998cafe368017df617bd1a120a24e WatchSource:0}: Error finding container fe1b4b813c7c6f3a1c289c5f01c66784889998cafe368017df617bd1a120a24e: Status 404 returned error can't find the container with id fe1b4b813c7c6f3a1c289c5f01c66784889998cafe368017df617bd1a120a24e
Sep 17 20:06:00.010858 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:00.010758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" event={"ID":"e8ab23dd-cc9d-41e4-9682-9df5e5584230","Type":"ContainerStarted","Data":"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"}
Sep 17 20:06:00.010858 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:00.010815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" event={"ID":"e8ab23dd-cc9d-41e4-9682-9df5e5584230","Type":"ContainerStarted","Data":"fe1b4b813c7c6f3a1c289c5f01c66784889998cafe368017df617bd1a120a24e"}
Sep 17 20:06:00.011052 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:00.010949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:06:00.029222 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:00.029172    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podStartSLOduration=1.029154779 podStartE2EDuration="1.029154779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:06:00.027777667 +0000 UTC m=+3592.041347829" watchObservedRunningTime="2026-09-17 20:06:00.029154779 +0000 UTC m=+3592.042724940"
Sep 17 20:06:10.390663 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:10.390629    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 20:06:10.401068 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:10.401045    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 20:06:31.017356 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:31.017323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:06:32.788358 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:32.788317    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:06:39.579808 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.579753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:06:39.583227 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.583209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.586595 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.586546    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-s6zwk\""
Sep 17 20:06:39.586595 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.586590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-q58dp\""
Sep 17 20:06:39.586955 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.586591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 20:06:39.586955 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.586724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 20:06:39.586955 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.586815    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 20:06:39.592823 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.592790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:06:39.663024 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.662988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.663024 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.663031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrzl\" (UniqueName: \"kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.764484 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.764450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.764484 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.764491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrzl\" (UniqueName: \"kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.767354 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.767322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.774863 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.774835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrzl\" (UniqueName: \"kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:39.895176 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:39.895071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:40.169147 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:40.169120    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:06:40.171301 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:06:40.171275    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae6849b_626d_49b8_ac2b_2bc6e15912f0.slice/crio-6b46daf34bb9924bddfa9b83ff0f495d61ce88cc2e149ba26d09b6d3702cc6ea WatchSource:0}: Error finding container 6b46daf34bb9924bddfa9b83ff0f495d61ce88cc2e149ba26d09b6d3702cc6ea: Status 404 returned error can't find the container with id 6b46daf34bb9924bddfa9b83ff0f495d61ce88cc2e149ba26d09b6d3702cc6ea
Sep 17 20:06:40.191994 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:40.191964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" event={"ID":"0ae6849b-626d-49b8-ac2b-2bc6e15912f0","Type":"ContainerStarted","Data":"6b46daf34bb9924bddfa9b83ff0f495d61ce88cc2e149ba26d09b6d3702cc6ea"}
Sep 17 20:06:41.017154 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:41.017116    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:06:41.199371 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:41.199338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" event={"ID":"0ae6849b-626d-49b8-ac2b-2bc6e15912f0","Type":"ContainerStarted","Data":"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"}
Sep 17 20:06:41.199549 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:41.199410    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:06:41.217794 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:41.217655    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podStartSLOduration=2.217615554 podStartE2EDuration="2.217615554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:06:41.216058587 +0000 UTC m=+3633.229628749" watchObservedRunningTime="2026-09-17 20:06:41.217615554 +0000 UTC m=+3633.231185715"
Sep 17 20:06:47.815809 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:47.815693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main" containerID="cri-o://63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761" gracePeriod=45
Sep 17 20:06:51.017125 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:06:51.017080    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:01.017321 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:01.017285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:01.017747 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:01.017382    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:07:11.017910 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:11.017872    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:12.205333 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:12.205301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:07:13.751011 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:13.750975    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:07:18.068610 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.068584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:07:18.087999 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.087925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx\") pod \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") "
Sep 17 20:07:18.087999 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.087968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs\") pod \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\" (UID: \"e8ab23dd-cc9d-41e4-9682-9df5e5584230\") "
Sep 17 20:07:18.090613 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.090581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx" (OuterVolumeSpecName: "kube-api-access-w69hx") pod "e8ab23dd-cc9d-41e4-9682-9df5e5584230" (UID: "e8ab23dd-cc9d-41e4-9682-9df5e5584230"). InnerVolumeSpecName "kube-api-access-w69hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:07:18.090613 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.090598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8ab23dd-cc9d-41e4-9682-9df5e5584230" (UID: "e8ab23dd-cc9d-41e4-9682-9df5e5584230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:07:18.189076 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.189036    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69hx\" (UniqueName: \"kubernetes.io/projected/e8ab23dd-cc9d-41e4-9682-9df5e5584230-kube-api-access-w69hx\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:07:18.189076 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.189069    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ab23dd-cc9d-41e4-9682-9df5e5584230-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:07:18.369491 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.369380    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerID="63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761" exitCode=0
Sep 17 20:07:18.369491 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.369441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"
Sep 17 20:07:18.369491 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.369474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" event={"ID":"e8ab23dd-cc9d-41e4-9682-9df5e5584230","Type":"ContainerDied","Data":"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"}
Sep 17 20:07:18.369886 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.369537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp" event={"ID":"e8ab23dd-cc9d-41e4-9682-9df5e5584230","Type":"ContainerDied","Data":"fe1b4b813c7c6f3a1c289c5f01c66784889998cafe368017df617bd1a120a24e"}
Sep 17 20:07:18.369886 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.369561    2581 scope.go:117] "RemoveContainer" containerID="63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"
Sep 17 20:07:18.378208 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.378189    2581 scope.go:117] "RemoveContainer" containerID="63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"
Sep 17 20:07:18.378543 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:07:18.378521    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761\": container with ID starting with 63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761 not found: ID does not exist" containerID="63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"
Sep 17 20:07:18.378616 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.378551    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761"} err="failed to get container status \"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761\": rpc error: code = NotFound desc = could not find container \"63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761\": container with ID starting with 63e52564e5e12c980ff10283aa2114a09ed96d41d37972466e923f89ddace761 not found: ID does not exist"
Sep 17 20:07:18.385206 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.385182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:07:18.390702 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:18.390681    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb8zlbp"]
Sep 17 20:07:20.137232 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:20.137149    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" path="/var/lib/kubelet/pods/e8ab23dd-cc9d-41e4-9682-9df5e5584230/volumes"
Sep 17 20:07:22.206026 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:22.205980    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:28.779150 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:28.779028    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main" containerID="cri-o://a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117" gracePeriod=45
Sep 17 20:07:32.205390 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:32.205350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:42.205685 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:42.205648    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:42.206063 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:42.205756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:07:48.711367 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.711328    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:07:48.711901 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.711688    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main"
Sep 17 20:07:48.711901 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.711701    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main"
Sep 17 20:07:48.711901 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.711773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ab23dd-cc9d-41e4-9682-9df5e5584230" containerName="main"
Sep 17 20:07:48.714554 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.714535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.717940 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.717919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 20:07:48.718356 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.718334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 20:07:48.718473 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.718387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-qtr79\""
Sep 17 20:07:48.718473 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.718334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-cbb96\""
Sep 17 20:07:48.718473 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.718336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 20:07:48.723704 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.723685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:07:48.850617 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.850585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j742r\" (UniqueName: \"kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.850805 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.850646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.951782 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.951742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.951972 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.951848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j742r\" (UniqueName: \"kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.954583 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.954556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:48.960340 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:48.960306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j742r\" (UniqueName: \"kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:49.025786 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.025695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:49.311937 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.311910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:07:49.314331 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:07:49.314302    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cfb333_ca4a_48a8_984a_c08832b0e25f.slice/crio-2b213720b29b1388cccb84ae9245bde521e0485297b4ce81a211e0245aa8785d WatchSource:0}: Error finding container 2b213720b29b1388cccb84ae9245bde521e0485297b4ce81a211e0245aa8785d: Status 404 returned error can't find the container with id 2b213720b29b1388cccb84ae9245bde521e0485297b4ce81a211e0245aa8785d
Sep 17 20:07:49.511580 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.511547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" event={"ID":"e3cfb333-ca4a-48a8-984a-c08832b0e25f","Type":"ContainerStarted","Data":"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"}
Sep 17 20:07:49.511824 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.511603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" event={"ID":"e3cfb333-ca4a-48a8-984a-c08832b0e25f","Type":"ContainerStarted","Data":"2b213720b29b1388cccb84ae9245bde521e0485297b4ce81a211e0245aa8785d"}
Sep 17 20:07:49.511824 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.511633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:07:49.535826 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:49.535778    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podStartSLOduration=1.535762504 podStartE2EDuration="1.535762504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:07:49.534002584 +0000 UTC m=+3701.547572745" watchObservedRunningTime="2026-09-17 20:07:49.535762504 +0000 UTC m=+3701.549332664"
Sep 17 20:07:52.205546 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:52.205507    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:07:59.035029 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.035004    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:07:59.139214 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.139113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs\") pod \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") "
Sep 17 20:07:59.139394 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.139236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clrzl\" (UniqueName: \"kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl\") pod \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\" (UID: \"0ae6849b-626d-49b8-ac2b-2bc6e15912f0\") "
Sep 17 20:07:59.141664 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.141640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl" (OuterVolumeSpecName: "kube-api-access-clrzl") pod "0ae6849b-626d-49b8-ac2b-2bc6e15912f0" (UID: "0ae6849b-626d-49b8-ac2b-2bc6e15912f0"). InnerVolumeSpecName "kube-api-access-clrzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:07:59.141727 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.141716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ae6849b-626d-49b8-ac2b-2bc6e15912f0" (UID: "0ae6849b-626d-49b8-ac2b-2bc6e15912f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:07:59.240548 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.240508    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrzl\" (UniqueName: \"kubernetes.io/projected/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-kube-api-access-clrzl\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:07:59.240548 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.240541    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae6849b-626d-49b8-ac2b-2bc6e15912f0-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:07:59.560159 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.560125    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerID="a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117" exitCode=0
Sep 17 20:07:59.560343 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.560185    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"
Sep 17 20:07:59.560343 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.560210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" event={"ID":"0ae6849b-626d-49b8-ac2b-2bc6e15912f0","Type":"ContainerDied","Data":"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"}
Sep 17 20:07:59.560343 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.560292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v" event={"ID":"0ae6849b-626d-49b8-ac2b-2bc6e15912f0","Type":"ContainerDied","Data":"6b46daf34bb9924bddfa9b83ff0f495d61ce88cc2e149ba26d09b6d3702cc6ea"}
Sep 17 20:07:59.560343 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.560316    2581 scope.go:117] "RemoveContainer" containerID="a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"
Sep 17 20:07:59.570034 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.570011    2581 scope.go:117] "RemoveContainer" containerID="a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"
Sep 17 20:07:59.570342 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:07:59.570321    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117\": container with ID starting with a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117 not found: ID does not exist" containerID="a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"
Sep 17 20:07:59.570438 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.570358    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117"} err="failed to get container status \"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117\": rpc error: code = NotFound desc = could not find container \"a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117\": container with ID starting with a8431f24a90c7f16676fde82feced616622ff3370be763e86cf0b48c831ef117 not found: ID does not exist"
Sep 17 20:07:59.581614 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.581593    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:07:59.585456 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:07:59.585434    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelxt7v"]
Sep 17 20:08:00.137806 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:00.137770    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" path="/var/lib/kubelet/pods/0ae6849b-626d-49b8-ac2b-2bc6e15912f0/volumes"
Sep 17 20:08:20.518368 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:20.518336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:08:22.341079 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:22.341040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:08:30.518587 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:30.518546    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:08:37.361283 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:37.361221    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main" containerID="cri-o://67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a" gracePeriod=45
Sep 17 20:08:40.517960 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:40.517919    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:08:50.517707 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:50.517615    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:08:50.518119 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:08:50.517730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:09:00.518239 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:00.518199    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 20:09:07.615248 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.615224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:09:07.708056 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.708030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs\") pod \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") "
Sep 17 20:09:07.708215 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.708066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j742r\" (UniqueName: \"kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r\") pod \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\" (UID: \"e3cfb333-ca4a-48a8-984a-c08832b0e25f\") "
Sep 17 20:09:07.710574 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.710543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3cfb333-ca4a-48a8-984a-c08832b0e25f" (UID: "e3cfb333-ca4a-48a8-984a-c08832b0e25f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:09:07.710688 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.710582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r" (OuterVolumeSpecName: "kube-api-access-j742r") pod "e3cfb333-ca4a-48a8-984a-c08832b0e25f" (UID: "e3cfb333-ca4a-48a8-984a-c08832b0e25f"). InnerVolumeSpecName "kube-api-access-j742r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:09:07.809295 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.809259    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3cfb333-ca4a-48a8-984a-c08832b0e25f-tls-certs\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:09:07.809295 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.809293    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j742r\" (UniqueName: \"kubernetes.io/projected/e3cfb333-ca4a-48a8-984a-c08832b0e25f-kube-api-access-j742r\") on node \"ip-10-0-132-57.ec2.internal\" DevicePath \"\""
Sep 17 20:09:07.861150 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.861121    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerID="67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a" exitCode=0
Sep 17 20:09:07.861294 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.861168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"
Sep 17 20:09:07.861294 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.861212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" event={"ID":"e3cfb333-ca4a-48a8-984a-c08832b0e25f","Type":"ContainerDied","Data":"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"}
Sep 17 20:09:07.861294 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.861264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm" event={"ID":"e3cfb333-ca4a-48a8-984a-c08832b0e25f","Type":"ContainerDied","Data":"2b213720b29b1388cccb84ae9245bde521e0485297b4ce81a211e0245aa8785d"}
Sep 17 20:09:07.861294 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.861287    2581 scope.go:117] "RemoveContainer" containerID="67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"
Sep 17 20:09:07.870325 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.870304    2581 scope.go:117] "RemoveContainer" containerID="67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"
Sep 17 20:09:07.870621 ip-10-0-132-57 kubenswrapper[2581]: E0917 20:09:07.870602    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a\": container with ID starting with 67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a not found: ID does not exist" containerID="67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"
Sep 17 20:09:07.870684 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.870632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a"} err="failed to get container status \"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a\": rpc error: code = NotFound desc = could not find container \"67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a\": container with ID starting with 67cc3e905f9f9749c38cde3d1d7ef6a27a5b664e280d6336817b48ed1ac3458a not found: ID does not exist"
Sep 17 20:09:07.883017 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.882992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:09:07.888563 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:07.888541    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9jd7dm"]
Sep 17 20:09:08.136889 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:08.136807    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" path="/var/lib/kubelet/pods/e3cfb333-ca4a-48a8-984a-c08832b0e25f/volumes"
Sep 17 20:09:39.461010 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:39.460981    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-584d86cbf4-w6hd2_07801f2c-540e-4886-9e6b-61e74fabf220/router/0.log"
Sep 17 20:09:40.362628 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:40.362601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-584d86cbf4-w6hd2_07801f2c-540e-4886-9e6b-61e74fabf220/router/0.log"
Sep 17 20:09:41.376154 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:41.376125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-t22dg_b1e280b9-d773-47c2-8daf-fdc28f7688df/authorino/0.log"
Sep 17 20:09:48.385138 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:48.385108    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gxwcj_8c5cff5c-f7d6-4184-90a8-7720b5051ba9/global-pull-secret-syncer/0.log"
Sep 17 20:09:48.550909 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:48.550883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6qz84_80ccbf37-d2de-40d3-89f8-0756439a2d4a/konnectivity-agent/0.log"
Sep 17 20:09:48.598031 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:48.598007    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-57.ec2.internal_03b66c5394879f7702479f341fee142c/haproxy/0.log"
Sep 17 20:09:53.564655 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:53.564572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-t22dg_b1e280b9-d773-47c2-8daf-fdc28f7688df/authorino/0.log"
Sep 17 20:09:55.290436 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:55.290384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-7pxn9_5e34a88e-3874-4271-a620-fc63f5c242c5/cluster-monitoring-operator/0.log"
Sep 17 20:09:55.486056 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:55.486020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl6s8_71e6adf5-5412-4d50-96e8-bc32c04ce607/node-exporter/0.log"
Sep 17 20:09:55.508191 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:55.508167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl6s8_71e6adf5-5412-4d50-96e8-bc32c04ce607/kube-rbac-proxy/0.log"
Sep 17 20:09:55.531812 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:55.531795    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kl6s8_71e6adf5-5412-4d50-96e8-bc32c04ce607/init-textfile/0.log"
Sep 17 20:09:56.108212 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:56.108172    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-z2txv_ec453613-ef16-4c60-9f0c-25dbb16eec5b/prometheus-operator-admission-webhook/0.log"
Sep 17 20:09:57.636436 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:57.636386    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7qrhw_53fb5fd1-2275-4cf0-bd83-3f3f49b12ee9/networking-console-plugin/0.log"
Sep 17 20:09:58.120672 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.120640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/2.log"
Sep 17 20:09:58.124451 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.124431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-stjnd_f302149c-75e1-489b-8ed4-51ff1c642044/console-operator/3.log"
Sep 17 20:09:58.554554 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554522    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"]
Sep 17 20:09:58.554827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554814    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main"
Sep 17 20:09:58.554827 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554827    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main"
Sep 17 20:09:58.554924 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main"
Sep 17 20:09:58.554924 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554858    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main"
Sep 17 20:09:58.554924 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554909    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cfb333-ca4a-48a8-984a-c08832b0e25f" containerName="main"
Sep 17 20:09:58.554924 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.554919    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae6849b-626d-49b8-ac2b-2bc6e15912f0" containerName="main"
Sep 17 20:09:58.557838 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.557819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.560644 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.560616    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhr8t\"/\"openshift-service-ca.crt\""
Sep 17 20:09:58.561437 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.561389    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhr8t\"/\"kube-root-ca.crt\""
Sep 17 20:09:58.561690 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.561669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rhr8t\"/\"default-dockercfg-cks9b\""
Sep 17 20:09:58.567754 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.567729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"]
Sep 17 20:09:58.611515 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.611484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnbp\" (UniqueName: \"kubernetes.io/projected/7addce13-120a-43b1-bb0b-9b844c9112c8-kube-api-access-9nnbp\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.611653 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.611535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-proc\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.611653 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.611633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-lib-modules\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.611726 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.611656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-sys\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.611726 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.611700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-podres\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.712624 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnbp\" (UniqueName: \"kubernetes.io/projected/7addce13-120a-43b1-bb0b-9b844c9112c8-kube-api-access-9nnbp\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-proc\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-lib-modules\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-sys\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-proc\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-podres\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-sys\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-podres\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.713058 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.712878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7addce13-120a-43b1-bb0b-9b844c9112c8-lib-modules\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.721613 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.721587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnbp\" (UniqueName: \"kubernetes.io/projected/7addce13-120a-43b1-bb0b-9b844c9112c8-kube-api-access-9nnbp\") pod \"perf-node-gather-daemonset-jqv4t\" (UID: \"7addce13-120a-43b1-bb0b-9b844c9112c8\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:58.876347 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:58.876279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:09:59.146342 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:59.146314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"]
Sep 17 20:09:59.148988 ip-10-0-132-57 kubenswrapper[2581]: W0917 20:09:59.148961    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7addce13_120a_43b1_bb0b_9b844c9112c8.slice/crio-7d1cc0124df0aaa22590c64389d138b2988580345bd6c16b654c8e90f501bf34 WatchSource:0}: Error finding container 7d1cc0124df0aaa22590c64389d138b2988580345bd6c16b654c8e90f501bf34: Status 404 returned error can't find the container with id 7d1cc0124df0aaa22590c64389d138b2988580345bd6c16b654c8e90f501bf34
Sep 17 20:09:59.243330 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:09:59.243304    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-49r8s_183b86fe-09a1-430e-9d15-e2fa1057d6f2/volume-data-source-validator/0.log"
Sep 17 20:10:00.091266 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.091234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t" event={"ID":"7addce13-120a-43b1-bb0b-9b844c9112c8","Type":"ContainerStarted","Data":"fa4011aedfb609594f45f2030c48916bd565da7793319524c4d23bac54cdc39d"}
Sep 17 20:10:00.091694 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.091284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t" event={"ID":"7addce13-120a-43b1-bb0b-9b844c9112c8","Type":"ContainerStarted","Data":"7d1cc0124df0aaa22590c64389d138b2988580345bd6c16b654c8e90f501bf34"}
Sep 17 20:10:00.091694 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.091312    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:10:00.107360 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.107304    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t" podStartSLOduration=2.107286345 podStartE2EDuration="2.107286345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:10:00.10663338 +0000 UTC m=+3832.120203540" watchObservedRunningTime="2026-09-17 20:10:00.107286345 +0000 UTC m=+3832.120856538"
Sep 17 20:10:00.300240 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.300211    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjfbc_5f282270-b758-4a96-9392-fa511f2059cb/dns/0.log"
Sep 17 20:10:00.322103 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.322070    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjfbc_5f282270-b758-4a96-9392-fa511f2059cb/kube-rbac-proxy/0.log"
Sep 17 20:10:00.427569 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:00.427543    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r29f5_051aa77a-c712-4c78-bb4b-e536d73b98f1/dns-node-resolver/0.log"
Sep 17 20:10:01.074696 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:01.074666    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v59rs_4abc53bb-f467-46c1-a12b-cc654a5e2a11/node-ca/0.log"
Sep 17 20:10:02.226908 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:02.226876    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-584d86cbf4-w6hd2_07801f2c-540e-4886-9e6b-61e74fabf220/router/0.log"
Sep 17 20:10:02.823524 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:02.823496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xdvl6_7860d54a-ff5f-4340-bca1-642b2736dc57/serve-healthcheck-canary/0.log"
Sep 17 20:10:03.300686 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:03.300650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-slzkf_a2cad30b-602b-41e5-a0d4-ff853e2973ed/insights-operator/0.log"
Sep 17 20:10:03.301653 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:03.301634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-slzkf_a2cad30b-602b-41e5-a0d4-ff853e2973ed/insights-operator/1.log"
Sep 17 20:10:03.397734 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:03.397707    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jzgh_e9968eda-be9c-48e1-b82f-9f3519b726cb/kube-rbac-proxy/0.log"
Sep 17 20:10:03.422439 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:03.422392    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jzgh_e9968eda-be9c-48e1-b82f-9f3519b726cb/exporter/0.log"
Sep 17 20:10:03.445776 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:03.445742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jzgh_e9968eda-be9c-48e1-b82f-9f3519b726cb/extractor/0.log"
Sep 17 20:10:06.106371 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:06.106343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-jqv4t"
Sep 17 20:10:06.495736 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:06.495707    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56c5b7c858-slnkm_b4d2f01c-ef24-4a43-a70a-921caf559235/manager/0.log"
Sep 17 20:10:07.929175 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:07.929144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-584d86cbf4-w6hd2_07801f2c-540e-4886-9e6b-61e74fabf220/router/0.log"
Sep 17 20:10:08.115432 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:08.115390    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-rgdjp_142d0912-1c13-4a0d-a83a-959fa5ea2012/manager/0.log"
Sep 17 20:10:09.229241 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:09.229214    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-584d86cbf4-w6hd2_07801f2c-540e-4886-9e6b-61e74fabf220/router/0.log"
Sep 17 20:10:13.407683 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:13.407652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9vn7w_2bc1564e-c179-4312-a165-8d68df6075e5/migrator/0.log"
Sep 17 20:10:13.434916 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:13.434888    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9vn7w_2bc1564e-c179-4312-a165-8d68df6075e5/graceful-termination/0.log"
Sep 17 20:10:13.795342 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:13.795310    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-hfq2j_c2b2f1a3-3474-4579-9259-8db6e55ddbf8/kube-storage-version-migrator-operator/1.log"
Sep 17 20:10:13.796570 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:13.796539    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-hfq2j_c2b2f1a3-3474-4579-9259-8db6e55ddbf8/kube-storage-version-migrator-operator/0.log"
Sep 17 20:10:15.001183 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.001101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/kube-multus-additional-cni-plugins/0.log"
Sep 17 20:10:15.029236 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.029204    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/egress-router-binary-copy/0.log"
Sep 17 20:10:15.052036 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.052010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/cni-plugins/0.log"
Sep 17 20:10:15.078654 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.078627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/bond-cni-plugin/0.log"
Sep 17 20:10:15.134010 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.133982    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/routeoverride-cni/0.log"
Sep 17 20:10:15.178878 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.178851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/whereabouts-cni-bincopy/0.log"
Sep 17 20:10:15.236464 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.236407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ptw2z_152784a6-5c9f-4a87-8a4f-873c218f9f37/whereabouts-cni/0.log"
Sep 17 20:10:15.538285 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.538256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v8h59_6d6199c3-283b-463c-98dc-6884c1d0ebfd/kube-multus/0.log"
Sep 17 20:10:15.665183 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.665153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wgtr9_41df1f9f-c14e-41fb-bf22-ae8346861773/network-metrics-daemon/0.log"
Sep 17 20:10:15.688594 ip-10-0-132-57 kubenswrapper[2581]: I0917 20:10:15.688568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wgtr9_41df1f9f-c14e-41fb-bf22-ae8346861773/kube-rbac-proxy/0.log"