Sep 18 19:14:26.290097 ip-10-0-143-75 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:14:29.645444 ip-10-0-143-75 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.645444 ip-10-0-143-75 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:14:29.645444 ip-10-0-143-75 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.645444 ip-10-0-143-75 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:14:29.645444 ip-10-0-143-75 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.646126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.645516    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:14:29.650144 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650129    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.650144 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650144    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650147    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650153    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650157    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650160    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650163    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650166    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650168    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650171    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650173    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650178    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650181    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650184    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650187    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650191    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650194    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650197    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650200    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650202    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.650210 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650205    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650208    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650210    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650213    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650217    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650220    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650223    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650227    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650230    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650232    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650235    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650237    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650240    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650243    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650246    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650250    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650253    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650256    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650258    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650261    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.650688 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650263    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650266    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650268    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650271    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650273    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650276    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650279    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650281    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650284    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650286    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650288    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650291    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650293    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650296    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650299    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650302    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650304    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650307    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650310    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650312    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.651211 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650315    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650317    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650319    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650323    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650326    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650328    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650331    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650334    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650336    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650339    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650341    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650344    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650348    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650351    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650355    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650358    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650361    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650363    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650366    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.651686 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650369    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650371    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650374    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650376    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650378    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650381    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650383    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650388    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650390    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650393    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650395    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650398    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650401    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650403    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650406    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650409    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650411    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650414    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650416    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.650418    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.652170 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651123    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651130    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651133    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651135    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651138    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651141    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651143    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651147    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651149    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651152    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651154    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651157    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651159    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651162    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651165    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651167    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651170    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651173    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651175    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651179    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.652700 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651182    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651184    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651187    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651189    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651192    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651194    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651197    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651200    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651202    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651204    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651207    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651209    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651211    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651214    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651217    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651219    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651222    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651226    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651230    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.653287 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651232    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651235    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651238    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651240    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651243    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651245    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651248    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651251    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651253    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651257    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651259    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651262    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651266    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651268    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651273    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651277    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651280    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651283    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651286    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651289    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.653772 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651291    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651294    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651297    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651299    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651302    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651304    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651307    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651309    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651312    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651314    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651317    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651320    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651322    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651325    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651328    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651330    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651333    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651335    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651338    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651340    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.654265 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651343    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651346    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651348    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651351    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651354    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651357    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651359    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651362    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651365    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651367    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651369    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651372    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651374    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651376    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651381    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651384    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651388    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651392    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651394    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.654766 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651397    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651788    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651794    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651797    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651800    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651803    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651805    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651808    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651811    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651814    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651817    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651821    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651824    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651828    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651831    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651835    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651839    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651842    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651846    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.655232 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651849    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651852    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651854    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651857    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651860    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651862    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651865    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651867    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651870    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651872    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651875    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651877    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651879    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651882    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651884    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651887    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651889    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651892    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651895    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651897    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.655691 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651900    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651902    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651905    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651908    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651911    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651913    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651915    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651919    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651921    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651924    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651927    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651929    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651932    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651934    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651936    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651939    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651941    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651944    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651946    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651949    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.656203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651951    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651953    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651956    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651958    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651961    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651963    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651966    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651968    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651970    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651973    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651978    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651981    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651984    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651986    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651989    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651992    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651994    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651997    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.651999    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652002    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.656715 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652005    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652007    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652010    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652013    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652015    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652018    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652020    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652023    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652025    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652027    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652030    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652032    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652034    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652037    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652039    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652042    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652044    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652046    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652049    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652052    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.657234 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652054    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652121    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652156    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652162    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652166    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652170    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652176    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652180    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652184    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652187    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652190    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652193    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652196    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652200    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652204    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652206    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652210    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652215    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652218    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652221    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652224    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652227    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652230    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652233    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:14:29.657710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652237    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652239    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652242    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652245    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652248    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652251    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652254    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652257    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652260    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652263    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652266    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652269    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652272    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652274    2582 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652278    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652282    2582 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652285    2582 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652288    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652291    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652305    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652309    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652314    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652317    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652320    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652323    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:14:29.658303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652326    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652329    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652332    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652334    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652337    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652340    2582 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652343    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652346    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652349    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652352    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652355    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652358    2582 flags.go:64] FLAG: --help="false"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652360    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-143-75.ec2.internal"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652363    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652366    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652369    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652372    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652375    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652378    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652381    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652383    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652386    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652389    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652392    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:14:29.659013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652395    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652399    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652401    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652404    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652408    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652411    2582 flags.go:64] FLAG: --lock-file=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652414    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652417    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652420    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652426    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652429    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652431    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652434    2582 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652437    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652440    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652443    2582 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652446    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652450    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652453    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652457    2582 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652460    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652462    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652465    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652468    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652471    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:14:29.659834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652474    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652477    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652484    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652486    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652490    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652493    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652495    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652501    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652504    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652507    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652510    2582 flags.go:64] FLAG: --port="10250"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652514    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652516    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03ed71e4ab036c076"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652519    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652523    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652526    2582 flags.go:64] FLAG: --register-node="true"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652529    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652532    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652535    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652538    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652540    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652546    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652549    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652552    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652555    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:14:29.660710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652557    2582 flags.go:64] FLAG: --runonce="false"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652560    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652563    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652566    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652568    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652571    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652574    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652577    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652580    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652582    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652585    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652588    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652590    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652593    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652596    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652599    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652604    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652606    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652610    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652614    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652616    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652619    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652622    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652625    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652628    2582 flags.go:64] FLAG: --v="2"
Sep 18 19:14:29.661346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652631    2582 flags.go:64] FLAG: --version="false"
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652635    2582 flags.go:64] FLAG: --vmodule=""
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652638    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.652641    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652767    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652770    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652774    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652777    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652780    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652782    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652785    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652787    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652790    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652792    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652796    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652800    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652803    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652805    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652808    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652812    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.661982 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652815    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652818    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652821    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652823    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652826    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652829    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652832    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652834    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652837    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652839    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652843    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652846    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652848    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652851    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652853    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652855    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652858    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652860    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652863    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652865    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.662469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652868    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652871    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652873    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652876    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652878    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652880    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652883    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652885    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652888    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652891    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652893    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652897    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652899    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652902    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652904    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652907    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652910    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652912    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652915    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.663038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652918    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652921    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652923    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652926    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652929    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652933    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652937    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652940    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652944    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652949    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652952    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652955    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652957    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652960    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652963    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652966    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652970    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652974    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652977    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.663509 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652979    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652982    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652984    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652986    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652989    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652993    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652996    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.652998    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653001    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653004    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653006    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653009    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653012    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653014    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653017    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653019    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653022    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653024    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653027    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653030    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.663995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653033    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653035    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653038    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653040    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.653042    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.653538    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662074    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662089    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662109    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662115    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662170    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662176    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662179    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662182    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662185    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.664469 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662188    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662190    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662193    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662196    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662199    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662201    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662204    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662206    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662209    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662212    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662214    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662217    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662220    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662222    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662225    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662228    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662230    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662233    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662235    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662238    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.664854 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662240    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662244    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662248    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662251    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662254    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662256    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662259    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662262    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662265    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662268    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662271    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662274    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662277    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662280    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662282    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662285    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662288    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662290    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662293    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.665327 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662295    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662299    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662301    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662304    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662306    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662309    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662311    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662314    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662316    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662319    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662321    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662324    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662327    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662329    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662332    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662334    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662336    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662339    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662341    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.665821 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662344    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662346    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662349    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662351    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662354    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662357    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662360    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662362    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662365    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662367    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662370    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662372    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662375    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662377    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662380    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662382    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662386    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662388    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662390    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662393    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.666320 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662395    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662398    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662400    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662402    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662405    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662407    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662410    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662412    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662414    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662418    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662421    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662423    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662427    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662431    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662434    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.666855 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662437    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662442    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662546    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662551    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662554    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662557    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662560    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662562    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662565    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662567    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662570    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662573    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662575    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662578    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.667227 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662584    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662587    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662589    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662592    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662594    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662597    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662599    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662602    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662604    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662608    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662612    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662614    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662617    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662619    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662622    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662625    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662627    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662630    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662632    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.667572 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662635    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662637    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662640    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662643    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662645    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662648    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662650    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662652    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662655    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662657    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662660    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662662    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662665    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662667    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662670    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662673    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662675    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662678    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662680    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662683    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.668038 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662685    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662688    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662690    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662693    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662695    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662697    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662700    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662702    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662705    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662708    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662712    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662715    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662717    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662720    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662723    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662744    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662747    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662750    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662753    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662756    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.668522 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662759    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662761    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662764    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662767    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662769    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662772    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662774    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662778    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662781    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662783    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662786    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662788    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662791    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662793    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662795    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662798    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662800    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662803    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662805    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662808    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.669015 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662810    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662813    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662817    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662820    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662824    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662827    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662830    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:29.662833    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662838    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.662938    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.665036    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.666337    2582 server.go:1019] "Starting client certificate rotation"
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.666432    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:29.669489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.666464    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:29.669837 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.669373    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:29.670507 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.670490    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:29.674181 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.674165    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:14:29.676568 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.676555    2582 log.go:25] "Validated CRI v1 image API"
Sep 18 19:14:29.679460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.679445    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:14:29.681784 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.681761    2582 fs.go:135] Filesystem UUIDs: map[362db362-dd29-4dd8-bb0f-964e4430d6f6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c228d094-8023-41bf-a161-5a2fb032c773:/dev/nvme0n1p4]
Sep 18 19:14:29.681847 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.681784    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:14:29.686777 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.686651    2582 manager.go:216] Machine: {Timestamp:2026-09-18 19:14:29.686205823 +0000 UTC m=+1.923332800 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101420 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2daff79b2c03334904f3c72f8d1a35 SystemUUID:ec2daff7-9b2c-0333-4904-f3c72f8d1a35 BootID:0d0d6fbd-1d7f-4a68-85af-fd34a60af7b2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:96:63:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:96:63:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:91:85:89:53:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:14:29.686777 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.686768    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:14:29.686896 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.686835    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:14:29.688298 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.688274    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:14:29.688419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.688300    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-75.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:14:29.689585 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.689576    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:14:29.689633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.689588    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:14:29.689633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.689599    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:29.689633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.689610    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:29.689721 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.689681    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:14:29.692751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692738    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:29.692866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692857    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:14:29.692899 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692890    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:14:29.692926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692899    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:14:29.692926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692913    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:14:29.692926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692921    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:14:29.693021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.692928    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:14:29.693225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.693208    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:29.693225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.693229    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:29.699883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.699664    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:14:29.701863 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.701846    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:14:29.701943 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.701870    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:14:29.701943 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.701890    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:14:29.702040 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.702001    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:14:29.702075 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702049    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:29.702887 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702875    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702892    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702898    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702904    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702913    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702919    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702935    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702941    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702951    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:14:29.702960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702957    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:14:29.703203 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702983    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:14:29.703203 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.702989    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:14:29.707205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.707192    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:14:29.707205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.707204    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:14:29.710827 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.710813    2582 server.go:1268] "Started kubelet"
Sep 18 19:14:29.710924 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.710882    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:14:29.710988 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.710944    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:14:29.711050 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.710900    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:14:29.711519 ip-10-0-143-75 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:14:29.712132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.711615    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-75.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:14:29.712132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.712026    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:14:29.712308 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.712243    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:14:29.714584 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.714566    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:14:29.719306 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719291    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:14:29.719391 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719306    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:29.719391 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.719372    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:29.719882 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719864    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:14:29.719882 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719864    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:14:29.720017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719893    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:14:29.720017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719984    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:14:29.720017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.719990    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:14:29.720278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720259    2582 factory.go:55] Registering systemd factory
Sep 18 19:14:29.720278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720280    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:14:29.720452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720436    2582 factory.go:153] Registering CRI-O factory
Sep 18 19:14:29.720507 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720458    2582 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:14:29.720542 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720506    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:14:29.720542 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720524    2582 factory.go:103] Registering Raw factory
Sep 18 19:14:29.720542 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720539    2582 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:14:29.720973 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.720947    2582 manager.go:349] Starting recovery of all containers
Sep 18 19:14:29.723867 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.723839    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:14:29.724199 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.724177    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:14:29.756931 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.723809    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-75.ec2.internal.18d680359eb75dde  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-75.ec2.internal,UID:ip-10-0-143-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-75.ec2.internal,},FirstTimestamp:2026-09-18 19:14:29.71079011 +0000 UTC m=+1.947917088,LastTimestamp:2026-09-18 19:14:29.71079011 +0000 UTC m=+1.947917088,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-75.ec2.internal,}"
Sep 18 19:14:29.757152 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.757130    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pskq"
Sep 18 19:14:29.757224 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.757160    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pskq"
Sep 18 19:14:29.758512 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.758475    2582 manager.go:354] Recovery completed
Sep 18 19:14:29.762324 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.762310    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.764358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764344    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.764431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764375    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.764431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764390    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.764936 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764919    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:14:29.764936 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764933    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.765017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.764935    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:14:29.765017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.765007    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:29.765569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.765556    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.765616 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.765580    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.765616 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.765589    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.766025 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.766015    2582 policy_none.go:49] "None policy: Start"
Sep 18 19:14:29.766065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.766030    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:14:29.766065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.766040    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:14:29.766994 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.766983    2582 policy_none.go:47] "Start"
Sep 18 19:14:29.797424 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797302    2582 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:14:29.797513 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.797451    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:14:29.797513 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797464    2582 server.go:87] "Starting device plugin registration server"
Sep 18 19:14:29.797691 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797677    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:14:29.797918 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797702    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:14:29.797918 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797860    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:14:29.798236 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797940    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:14:29.798236 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.797949    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:14:29.798466 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.798437    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:14:29.798539 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.798490    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:29.887701 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.887675    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:14:29.888937 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.888916    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:14:29.889000 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.888959    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:14:29.889000 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.888981    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:14:29.889076 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.889018    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:14:29.893879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.893863    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:29.898580 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.898544    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.899275 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.899260    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.899348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.899287    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.899348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.899296    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.899348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.899320    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-75.ec2.internal"
Sep 18 19:14:29.909398 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.909385    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-75.ec2.internal"
Sep 18 19:14:29.909443 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.909405    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-75.ec2.internal\": node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:29.959109 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:29.959094    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:29.989784 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.989765    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"]
Sep 18 19:14:29.989824 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.989820    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.991437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.991419    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.991525 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.991449    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.991525 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.991458    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.992708 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.992695    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.992844 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.992826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:29.992899 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.992865    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.993386 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993371    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.993457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993395    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.993457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993409    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.993457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993431    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.993457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993452    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.993625 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.993465    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.994617 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.994604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:29.994673 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.994627    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.995260 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.995246    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.995325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.995271    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.995325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:29.995283    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:30.022116 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.022094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.022194 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.022118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e1249619055b5b051291a965f220aa-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.ec2.internal\" (UID: \"10e1249619055b5b051291a965f220aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.022194 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.022141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.026247 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.026227    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.ec2.internal\" not found" node="ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.030503 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.030489    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-75.ec2.internal\" not found" node="ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.059574 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.059554    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:30.123069 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.123171 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.123171 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.123171 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3599f1063f207d0364594c6f19dc0d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal\" (UID: \"c3599f1063f207d0364594c6f19dc0d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.123171 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e1249619055b5b051291a965f220aa-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.ec2.internal\" (UID: \"10e1249619055b5b051291a965f220aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.123171 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.123154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e1249619055b5b051291a965f220aa-config\") pod \"kube-apiserver-proxy-ip-10-0-143-75.ec2.internal\" (UID: \"10e1249619055b5b051291a965f220aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.160196 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.160152    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:30.260899 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.260871    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:30.329352 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.329328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.333843 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.333751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.361886 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.361862    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:30.462360 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.462298    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-75.ec2.internal\" not found"
Sep 18 19:14:30.541040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.541020    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:30.620132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.620108    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.657834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.657816    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:30.659901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.659888    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.665291 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.665275    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:30.665411 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.665392    2582 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://ad06820afe55e4cdaa45c8fa39f92d2a-a9f98b7f542b6f26.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.143.75:45464->18.235.169.19:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:30.665478 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.665401    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:30.693341 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.693327    2582 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:30.701505 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.701490    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:30.702667 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.702641    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7ts68","openshift-multus/multus-additional-cni-plugins-xrpn6","openshift-multus/network-metrics-daemon-2bqwp","openshift-ovn-kubernetes/ovnkube-node-wlzg5","kube-system/global-pull-secret-syncer-spxh5","kube-system/konnectivity-agent-znpkc","openshift-cluster-node-tuning-operator/tuned-mdxcq","openshift-network-diagnostics/network-check-target-phr2x","openshift-network-operator/iptables-alerter-g5r4z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5","openshift-dns/node-resolver-kbjz4","openshift-image-registry/node-ca-nwg8m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal"]
Sep 18 19:14:30.704652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.704627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.705831 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.705811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.707200 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.707184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.707255 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.707235    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:30.708886 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.708873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.709786 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.709708    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:30.709786 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.709762    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.710112 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.710097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.710177 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.710160    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:30.710569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.710556    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.710772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.710716    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6qc5d\""
Sep 18 19:14:30.711405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.711391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.712565 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.712529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.713559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.713545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:30.713619 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.713609    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:30.714801 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.714784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.716514 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.716497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.716605 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.716524    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:30.716770 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.716755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:30.716946 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.716923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.716946 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.716942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8l98r\""
Sep 18 19:14:30.717093 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.717046    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:30.718019 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.718002    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.718519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.718501    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.718808 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.718790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.718899 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.718791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.719411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.719395    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:30.719948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.719929    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:30.720064 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.720045    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.720134 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.720082    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:30.720464 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.720450    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:30.720612 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.720593    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:30.720807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.720792    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.726892 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.726875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-sys\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.726983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.726905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysconfig\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.726983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.726929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-iptables-alerter-script\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.726983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.726951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brbx\" (UniqueName: \"kubernetes.io/projected/f2423e66-5b43-469f-8788-45c860120f43-kube-api-access-5brbx\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.726998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-modprobe-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-lib-modules\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-var-lib-kubelet\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-dbus\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.727085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-device-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-bin\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-env-overrides\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslcv\" (UniqueName: \"kubernetes.io/projected/9917eb6e-c6ba-494c-b736-c205153c0e1c-kube-api-access-jslcv\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc468a-d53a-4092-ab1f-73139aea1307-tmp-dir\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-host-slash\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-conf\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727310 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-systemd-units\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-systemd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-log-socket\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727353    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-multus\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-hostroot\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-run\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-tmp\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-multus-daemon-config\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljffm\" (UniqueName: \"kubernetes.io/projected/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-kube-api-access-ljffm\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.727575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-kubelet\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-netns\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdfw\" (UniqueName: \"kubernetes.io/projected/6cc65196-8ae5-48cb-8e49-9e8847db616a-kube-api-access-7jdfw\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c34a65-61c3-418d-8599-87cb5547cba6-ovn-node-metrics-cert\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-multus-certs\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695dt\" (UniqueName: \"kubernetes.io/projected/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-kube-api-access-695dt\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-kubernetes\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-cni-binary-copy\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-kubelet\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cc65196-8ae5-48cb-8e49-9e8847db616a-host\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-ovn\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.728117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cc65196-8ae5-48cb-8e49-9e8847db616a-serviceca\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-systemd\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-host\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5c57212-561c-4172-9325-8d50d8b10155-konnectivity-ca\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-system-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.727994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-bin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-etc-kubernetes\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-socket-dir-parent\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-etc-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-script-lib\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mxp\" (UniqueName: \"kubernetes.io/projected/45c34a65-61c3-418d-8599-87cb5547cba6-kube-api-access-p6mxp\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9adc468a-d53a-4092-ab1f-73139aea1307-hosts-file\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmxd\" (UniqueName: \"kubernetes.io/projected/9adc468a-d53a-4092-ab1f-73139aea1307-kube-api-access-xnmxd\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-k8s-cni-cncf-io\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-tuned\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-node-log\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.728712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-netd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-config\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-kubelet-config\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5c57212-561c-4172-9325-8d50d8b10155-agent-certs\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cnibin\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frv8\" (UniqueName: \"kubernetes.io/projected/e7ddf57d-4978-4517-a42b-3d1bbd200461-kube-api-access-8frv8\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-var-lib-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8gg\" (UniqueName: \"kubernetes.io/projected/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-kube-api-access-8h8gg\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-netns\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-slash\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-cnibin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-conf-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.728508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729061    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729262    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r9mzq\""
Sep 18 19:14:30.729348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729281    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729293    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729387    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729589    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729653    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729657    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:30.730045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.729997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:30.730411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730090    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2n9bn\""
Sep 18 19:14:30.730411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tq9fm\""
Sep 18 19:14:30.730411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730328    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sjwl2\""
Sep 18 19:14:30.730411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f67ck\""
Sep 18 19:14:30.730660 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8xfg\""
Sep 18 19:14:30.730660 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.730619    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:30.733818 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.733792    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4h5m\""
Sep 18 19:14:30.761586 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.761550    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:29 +0000 UTC" deadline="2028-04-19 17:45:08.52040977 +0000 UTC"
Sep 18 19:14:30.761750 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.761586    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13894h30m37.758827306s"
Sep 18 19:14:30.783912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.783890    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:30.820825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.820806    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:30.829067 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cc65196-8ae5-48cb-8e49-9e8847db616a-serviceca\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.829159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-systemd\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-host\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5c57212-561c-4172-9325-8d50d8b10155-konnectivity-ca\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.829159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-system-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-systemd\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-bin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-host\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-etc-kubernetes\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-bin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-socket-dir-parent\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-etc-kubernetes\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-etc-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-system-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-socket-dir-parent\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-script-lib\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-etc-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mxp\" (UniqueName: \"kubernetes.io/projected/45c34a65-61c3-418d-8599-87cb5547cba6-kube-api-access-p6mxp\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9adc468a-d53a-4092-ab1f-73139aea1307-hosts-file\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.829397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmxd\" (UniqueName: \"kubernetes.io/projected/9adc468a-d53a-4092-ab1f-73139aea1307-kube-api-access-xnmxd\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-k8s-cni-cncf-io\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-tuned\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-node-log\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-netd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-config\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-kubelet-config\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cc65196-8ae5-48cb-8e49-9e8847db616a-serviceca\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5c57212-561c-4172-9325-8d50d8b10155-agent-certs\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9adc468a-d53a-4092-ab1f-73139aea1307-hosts-file\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-netd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cnibin\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-node-log\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cnibin\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frv8\" (UniqueName: \"kubernetes.io/projected/e7ddf57d-4978-4517-a42b-3d1bbd200461-kube-api-access-8frv8\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-k8s-cni-cncf-io\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.830277 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5c57212-561c-4172-9325-8d50d8b10155-konnectivity-ca\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-var-lib-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.829802    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8gg\" (UniqueName: \"kubernetes.io/projected/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-kube-api-access-8h8gg\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-netns\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.829864    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:31.329842624 +0000 UTC m=+3.566969602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.829874    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-slash\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829896    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-cnibin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.829927    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:14:31.329916178 +0000 UTC m=+3.567043174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-conf-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-netns\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829975    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-kubelet-config\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.831063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-var-lib-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-cnibin\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.829796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-script-lib\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-sys\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-conf-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-sys\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-slash\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysconfig\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-iptables-alerter-script\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brbx\" (UniqueName: \"kubernetes.io/projected/f2423e66-5b43-469f-8788-45c860120f43-kube-api-access-5brbx\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-modprobe-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-lib-modules\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-var-lib-kubelet\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-dbus\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.831898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-modprobe-d\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysconfig\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-var-lib-kubelet\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b36dced-2265-4d55-9ff6-e28b8102b04e-dbus\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-ovnkube-config\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-device-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-lib-modules\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-bin\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-device-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-multus-cni-dir\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-env-overrides\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-iptables-alerter-script\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-cni-bin\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.832654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslcv\" (UniqueName: \"kubernetes.io/projected/9917eb6e-c6ba-494c-b736-c205153c0e1c-kube-api-access-jslcv\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc468a-d53a-4092-ab1f-73139aea1307-tmp-dir\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-host-slash\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-conf\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-systemd-units\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-systemd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-log-socket\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-multus\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-hostroot\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.830996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-run\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc468a-d53a-4092-ab1f-73139aea1307-tmp-dir\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-systemd-units\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c34a65-61c3-418d-8599-87cb5547cba6-env-overrides\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-systemd\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-tmp\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-host-slash\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-multus-daemon-config\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.833154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-ovn-kubernetes\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-hostroot\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-sysctl-conf\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-run\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljffm\" (UniqueName: \"kubernetes.io/projected/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-kube-api-access-ljffm\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-kubelet\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-netns\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-cni-multus\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-log-socket\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-run-netns\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-host-kubelet\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.833912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdfw\" (UniqueName: \"kubernetes.io/projected/6cc65196-8ae5-48cb-8e49-9e8847db616a-kube-api-access-7jdfw\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c34a65-61c3-418d-8599-87cb5547cba6-ovn-node-metrics-cert\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-multus-certs\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695dt\" (UniqueName: \"kubernetes.io/projected/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-kube-api-access-695dt\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-kubernetes\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-cni-binary-copy\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-kubelet\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.831973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cc65196-8ae5-48cb-8e49-9e8847db616a-host\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-ovn\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.834757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9917eb6e-c6ba-494c-b736-c205153c0e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-multus-daemon-config\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cc65196-8ae5-48cb-8e49-9e8847db616a-host\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-var-lib-kubelet\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-openvswitch\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f2423e66-5b43-469f-8788-45c860120f43-host-run-multus-certs\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c34a65-61c3-418d-8599-87cb5547cba6-run-ovn\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-kubernetes\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.832565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2423e66-5b43-469f-8788-45c860120f43-cni-binary-copy\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.833674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-etc-tuned\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.833899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7ddf57d-4978-4517-a42b-3d1bbd200461-tmp\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.834173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5c57212-561c-4172-9325-8d50d8b10155-agent-certs\") pod \"konnectivity-agent-znpkc\" (UID: \"e5c57212-561c-4172-9325-8d50d8b10155\") " pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:30.835571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.834405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c34a65-61c3-418d-8599-87cb5547cba6-ovn-node-metrics-cert\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.854263 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.854248    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:30.860690 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.860670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmxd\" (UniqueName: \"kubernetes.io/projected/9adc468a-d53a-4092-ab1f-73139aea1307-kube-api-access-xnmxd\") pod \"node-resolver-kbjz4\" (UID: \"9adc468a-d53a-4092-ab1f-73139aea1307\") " pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:30.866645 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.866630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brbx\" (UniqueName: \"kubernetes.io/projected/f2423e66-5b43-469f-8788-45c860120f43-kube-api-access-5brbx\") pod \"multus-7ts68\" (UID: \"f2423e66-5b43-469f-8788-45c860120f43\") " pod="openshift-multus/multus-7ts68"
Sep 18 19:14:30.868542 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.868526    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:30.868644 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.868549    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:30.868644 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.868560    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:30.868644 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:30.868622    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:31.368603719 +0000 UTC m=+3.605730702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:30.869903 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.869881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslcv\" (UniqueName: \"kubernetes.io/projected/9917eb6e-c6ba-494c-b736-c205153c0e1c-kube-api-access-jslcv\") pod \"aws-ebs-csi-driver-node-cm9m5\" (UID: \"9917eb6e-c6ba-494c-b736-c205153c0e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:30.875364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.875349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdfw\" (UniqueName: \"kubernetes.io/projected/6cc65196-8ae5-48cb-8e49-9e8847db616a-kube-api-access-7jdfw\") pod \"node-ca-nwg8m\" (UID: \"6cc65196-8ae5-48cb-8e49-9e8847db616a\") " pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:30.886300 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.886280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljffm\" (UniqueName: \"kubernetes.io/projected/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-kube-api-access-ljffm\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:30.888715 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.888700    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrhwx"
Sep 18 19:14:30.889339 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.889323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-695dt\" (UniqueName: \"kubernetes.io/projected/41d787f6-5cce-4ebd-8d40-4a4eedd33ca4-kube-api-access-695dt\") pod \"multus-additional-cni-plugins-xrpn6\" (UID: \"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4\") " pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:30.892556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.892535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mxp\" (UniqueName: \"kubernetes.io/projected/45c34a65-61c3-418d-8599-87cb5547cba6-kube-api-access-p6mxp\") pod \"ovnkube-node-wlzg5\" (UID: \"45c34a65-61c3-418d-8599-87cb5547cba6\") " pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:30.895380 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:30.895352    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3599f1063f207d0364594c6f19dc0d1.slice/crio-acb5033b9c138911df8eaa5713f4e43bf738bb48ac70b65c892cff5ed53680fa WatchSource:0}: Error finding container acb5033b9c138911df8eaa5713f4e43bf738bb48ac70b65c892cff5ed53680fa: Status 404 returned error can't find the container with id acb5033b9c138911df8eaa5713f4e43bf738bb48ac70b65c892cff5ed53680fa
Sep 18 19:14:30.895718 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:30.895701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e1249619055b5b051291a965f220aa.slice/crio-5951125dc7853452b32665dfa057cb03af8997c94b916dc61d32ff7e1a850fd2 WatchSource:0}: Error finding container 5951125dc7853452b32665dfa057cb03af8997c94b916dc61d32ff7e1a850fd2: Status 404 returned error can't find the container with id 5951125dc7853452b32665dfa057cb03af8997c94b916dc61d32ff7e1a850fd2
Sep 18 19:14:30.899771 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.899748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frv8\" (UniqueName: \"kubernetes.io/projected/e7ddf57d-4978-4517-a42b-3d1bbd200461-kube-api-access-8frv8\") pod \"tuned-mdxcq\" (UID: \"e7ddf57d-4978-4517-a42b-3d1bbd200461\") " pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:30.903597 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.901344    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:30.904217 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.904200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8gg\" (UniqueName: \"kubernetes.io/projected/8d0b7331-c1a1-4c7f-a9ce-5fd466776677-kube-api-access-8h8gg\") pod \"iptables-alerter-g5r4z\" (UID: \"8d0b7331-c1a1-4c7f-a9ce-5fd466776677\") " pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:30.916565 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.916549    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrhwx"
Sep 18 19:14:30.953723 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:30.953694    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:31.038004 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.037959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwg8m"
Sep 18 19:14:31.044592 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.044573    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc65196_8ae5_48cb_8e49_9e8847db616a.slice/crio-3e3c3a1112c0cbd294e26eb255e4df0c337e24b34afc07281f2294764b547e3d WatchSource:0}: Error finding container 3e3c3a1112c0cbd294e26eb255e4df0c337e24b34afc07281f2294764b547e3d: Status 404 returned error can't find the container with id 3e3c3a1112c0cbd294e26eb255e4df0c337e24b34afc07281f2294764b547e3d
Sep 18 19:14:31.050643 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.050627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mdxcq"
Sep 18 19:14:31.057283 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.057261    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ddf57d_4978_4517_a42b_3d1bbd200461.slice/crio-6194ac44aa45e2eced4f93a53ab0774840b2cc956a60a0a15e8e1aec48151182 WatchSource:0}: Error finding container 6194ac44aa45e2eced4f93a53ab0774840b2cc956a60a0a15e8e1aec48151182: Status 404 returned error can't find the container with id 6194ac44aa45e2eced4f93a53ab0774840b2cc956a60a0a15e8e1aec48151182
Sep 18 19:14:31.059089 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.059042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:31.065576 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.065555    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c34a65_61c3_418d_8599_87cb5547cba6.slice/crio-6a7ae785ad055a482dc5b4aa25c2818a8a20f47c2c7b74f6c74c12f78c8520b7 WatchSource:0}: Error finding container 6a7ae785ad055a482dc5b4aa25c2818a8a20f47c2c7b74f6c74c12f78c8520b7: Status 404 returned error can't find the container with id 6a7ae785ad055a482dc5b4aa25c2818a8a20f47c2c7b74f6c74c12f78c8520b7
Sep 18 19:14:31.085523 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.085508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:31.093324 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.093304    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c57212_561c_4172_9325_8d50d8b10155.slice/crio-39081b88c5c4379fac2c91968422d368c6a39d1c13253f7b1b89bc511c92e0bc WatchSource:0}: Error finding container 39081b88c5c4379fac2c91968422d368c6a39d1c13253f7b1b89bc511c92e0bc: Status 404 returned error can't find the container with id 39081b88c5c4379fac2c91968422d368c6a39d1c13253f7b1b89bc511c92e0bc
Sep 18 19:14:31.099090 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.099071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ts68"
Sep 18 19:14:31.105205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.105191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5r4z"
Sep 18 19:14:31.105380 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.105361    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2423e66_5b43_469f_8788_45c860120f43.slice/crio-f99400c3324df36b93047e01d4edb59c8c7aff5d537529962c555c2a8ce3aeb1 WatchSource:0}: Error finding container f99400c3324df36b93047e01d4edb59c8c7aff5d537529962c555c2a8ce3aeb1: Status 404 returned error can't find the container with id f99400c3324df36b93047e01d4edb59c8c7aff5d537529962c555c2a8ce3aeb1
Sep 18 19:14:31.110373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.110355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5"
Sep 18 19:14:31.111584 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.111563    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0b7331_c1a1_4c7f_a9ce_5fd466776677.slice/crio-dbac0cf21bcf27b1f16e62c7d0f6c71cf70943cdf3bd336ea7dd4f669d8ac4cf WatchSource:0}: Error finding container dbac0cf21bcf27b1f16e62c7d0f6c71cf70943cdf3bd336ea7dd4f669d8ac4cf: Status 404 returned error can't find the container with id dbac0cf21bcf27b1f16e62c7d0f6c71cf70943cdf3bd336ea7dd4f669d8ac4cf
Sep 18 19:14:31.116984 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.116963    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9917eb6e_c6ba_494c_b736_c205153c0e1c.slice/crio-37fc68ef4130f912fe4298878c7600ad9d51c0f8eb68232896110253fac10913 WatchSource:0}: Error finding container 37fc68ef4130f912fe4298878c7600ad9d51c0f8eb68232896110253fac10913: Status 404 returned error can't find the container with id 37fc68ef4130f912fe4298878c7600ad9d51c0f8eb68232896110253fac10913
Sep 18 19:14:31.124552 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.124534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kbjz4"
Sep 18 19:14:31.128742 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.128706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xrpn6"
Sep 18 19:14:31.131473 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.131453    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc468a_d53a_4092_ab1f_73139aea1307.slice/crio-4c8a6cd752b2ca95faa87d39f17e0df9a9f1e14ee31f9c8b6512718607d13c84 WatchSource:0}: Error finding container 4c8a6cd752b2ca95faa87d39f17e0df9a9f1e14ee31f9c8b6512718607d13c84: Status 404 returned error can't find the container with id 4c8a6cd752b2ca95faa87d39f17e0df9a9f1e14ee31f9c8b6512718607d13c84
Sep 18 19:14:31.135917 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:14:31.135898    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d787f6_5cce_4ebd_8d40_4a4eedd33ca4.slice/crio-331797a87117e502adb92184d868b84ccdd9bf899a3da413117495c4dc00cf53 WatchSource:0}: Error finding container 331797a87117e502adb92184d868b84ccdd9bf899a3da413117495c4dc00cf53: Status 404 returned error can't find the container with id 331797a87117e502adb92184d868b84ccdd9bf899a3da413117495c4dc00cf53
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.335784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.335828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.335940    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.335990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.335976074 +0000 UTC m=+4.573103051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.336353    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:31.336446 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.336404    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.336389095 +0000 UTC m=+4.573516077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:31.436364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.436327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:31.436598 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.436530    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:31.436598 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.436552    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:31.436598 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.436564    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:31.436797 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:31.436618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.43660041 +0000 UTC m=+4.673727373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:31.902546 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.902411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"6a7ae785ad055a482dc5b4aa25c2818a8a20f47c2c7b74f6c74c12f78c8520b7"}
Sep 18 19:14:31.920322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.920244    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:30 +0000 UTC" deadline="2028-05-23 07:23:15.913292935 +0000 UTC"
Sep 18 19:14:31.920322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.920278    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14700h8m43.993019717s"
Sep 18 19:14:31.924965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.924888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwg8m" event={"ID":"6cc65196-8ae5-48cb-8e49-9e8847db616a","Type":"ContainerStarted","Data":"3e3c3a1112c0cbd294e26eb255e4df0c337e24b34afc07281f2294764b547e3d"}
Sep 18 19:14:31.934578 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.934481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kbjz4" event={"ID":"9adc468a-d53a-4092-ab1f-73139aea1307","Type":"ContainerStarted","Data":"4c8a6cd752b2ca95faa87d39f17e0df9a9f1e14ee31f9c8b6512718607d13c84"}
Sep 18 19:14:31.963878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.963846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ts68" event={"ID":"f2423e66-5b43-469f-8788-45c860120f43","Type":"ContainerStarted","Data":"f99400c3324df36b93047e01d4edb59c8c7aff5d537529962c555c2a8ce3aeb1"}
Sep 18 19:14:31.970273 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.970247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdxcq" event={"ID":"e7ddf57d-4978-4517-a42b-3d1bbd200461","Type":"ContainerStarted","Data":"6194ac44aa45e2eced4f93a53ab0774840b2cc956a60a0a15e8e1aec48151182"}
Sep 18 19:14:31.981919 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.981885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal" event={"ID":"10e1249619055b5b051291a965f220aa","Type":"ContainerStarted","Data":"5951125dc7853452b32665dfa057cb03af8997c94b916dc61d32ff7e1a850fd2"}
Sep 18 19:14:31.995815 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:31.995716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal" event={"ID":"c3599f1063f207d0364594c6f19dc0d1","Type":"ContainerStarted","Data":"acb5033b9c138911df8eaa5713f4e43bf738bb48ac70b65c892cff5ed53680fa"}
Sep 18 19:14:32.008339 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.008318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerStarted","Data":"331797a87117e502adb92184d868b84ccdd9bf899a3da413117495c4dc00cf53"}
Sep 18 19:14:32.026161 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.026135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5" event={"ID":"9917eb6e-c6ba-494c-b736-c205153c0e1c","Type":"ContainerStarted","Data":"37fc68ef4130f912fe4298878c7600ad9d51c0f8eb68232896110253fac10913"}
Sep 18 19:14:32.031240 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.031053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5r4z" event={"ID":"8d0b7331-c1a1-4c7f-a9ce-5fd466776677","Type":"ContainerStarted","Data":"dbac0cf21bcf27b1f16e62c7d0f6c71cf70943cdf3bd336ea7dd4f669d8ac4cf"}
Sep 18 19:14:32.039183 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.039159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znpkc" event={"ID":"e5c57212-561c-4172-9325-8d50d8b10155","Type":"ContainerStarted","Data":"39081b88c5c4379fac2c91968422d368c6a39d1c13253f7b1b89bc511c92e0bc"}
Sep 18 19:14:32.262533 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.262452    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:32.344917 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.344885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:32.345094 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.344931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:32.345094 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.345071    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:32.345203 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.345126    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.345108729 +0000 UTC m=+6.582235708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:32.345534 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.345512    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:32.345635 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.345571    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.345554056 +0000 UTC m=+6.582681023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:32.445654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.445617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:32.445877 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.445855    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:32.445970 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.445889    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:32.445970 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.445905    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:32.446067 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.445970    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:34.445941384 +0000 UTC m=+6.683068348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:32.889697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.889663    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:32.889895 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.889705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:32.889895 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.889808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:32.889895 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.889878    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:32.890064 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.889937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:32.890064 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:32.890010    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:32.920763 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.920715    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:30 +0000 UTC" deadline="2028-04-06 08:55:20.243839974 +0000 UTC"
Sep 18 19:14:32.920763 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:32.920765    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13573h40m47.323080975s"
Sep 18 19:14:34.362229 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.362193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:34.362663 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.362240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:34.362663 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.362385    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.362663 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.362438    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:14:38.362421663 +0000 UTC m=+10.599548633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:34.362999 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.362825    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.362999 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.362874    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:38.362859397 +0000 UTC m=+10.599986367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:34.463504 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.463465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:34.463677 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.463618    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:34.463677 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.463637    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:34.463677 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.463650    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.463857 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.463703    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:38.463686233 +0000 UTC m=+10.700813213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:34.889347 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.889313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:34.889546 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.889449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:34.889845 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.889824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:34.889956 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.889919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:34.890023 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:34.889968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:34.890075 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:34.890027    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:36.889250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:36.889215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:36.889699 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:36.889369    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:36.889699 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:36.889456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:36.889699 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:36.889527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:36.889699 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:36.889584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:36.889699 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:36.889657    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:38.399138 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.399096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:38.399565 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.399155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:38.399565 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.399328    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:38.399565 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.399390    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:14:46.399372196 +0000 UTC m=+18.636499160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:38.399874 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.399834    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:38.399969 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.399907    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:46.399890182 +0000 UTC m=+18.637017153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:38.499817 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.499771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:38.500068 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.500049    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:38.500150 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.500074    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:38.500150 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.500088    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:38.500150 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.500144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:46.500125411 +0000 UTC m=+18.737252377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.890074    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.890191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.890226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.890311    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:38.890075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:38.890526 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:38.890395    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:40.889693 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:40.889611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:40.890192 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:40.889611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:40.890192 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:40.889759    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:40.890192 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:40.889846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:40.890192 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:40.889611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:40.890192 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:40.889939    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:42.889965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:42.890069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:42.890092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:42.890084    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:42.890150    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:42.890395 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:42.890244    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:44.889810 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:44.889769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:44.890207 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:44.889769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:44.890207 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:44.890009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:44.890207 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:44.889898    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:44.890207 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:44.889769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:44.890207 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:44.890101    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:46.450991 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.450903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:46.450991 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.450945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:46.451416 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.451058    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:46.451416 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.451065    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:46.451416 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.451120    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:02.451102799 +0000 UTC m=+34.688229764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:46.451416 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.451136    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:02.451129044 +0000 UTC m=+34.688256008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:46.551805 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.551766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:46.551976 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.551924    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:46.551976 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.551943    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:46.551976 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.551957    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:46.552095 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.552018    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:02.55199902 +0000 UTC m=+34.789126022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:46.889572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.889541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:46.889572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.889572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:46.889904 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.889633    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:46.889904 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:46.889666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:46.889904 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.889766    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:46.889904 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:46.889826    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:48.889331 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:48.889300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:48.889331 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:48.889328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:48.889815 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:48.889341    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:48.889815 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:48.889405    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:48.889815 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:48.889467    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:48.889815 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:48.889606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:50.889880 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:50.889483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:50.890226 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:50.889993    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:50.890226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:50.889552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:50.890226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:50.889536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:50.890226 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:50.890082    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:50.890226 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:50.890163    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:51.090448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.090401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kbjz4" event={"ID":"9adc468a-d53a-4092-ab1f-73139aea1307","Type":"ContainerStarted","Data":"49d55d0d45471a9b8fb2766ebd13718aa5d5b45e693681f7c8e74faf0d8d1890"}
Sep 18 19:14:51.092076 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.092049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ts68" event={"ID":"f2423e66-5b43-469f-8788-45c860120f43","Type":"ContainerStarted","Data":"a42f57ee824eb6d4675341763f59f8db7a7f9c1cc295d021f4e3bf064d0046e8"}
Sep 18 19:14:51.093740 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.093698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mdxcq" event={"ID":"e7ddf57d-4978-4517-a42b-3d1bbd200461","Type":"ContainerStarted","Data":"3a4fce16f1a1b26c6a2f95f403a9b39e7aef2eba2bb3608ec792877c431c7173"}
Sep 18 19:14:51.095272 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.095250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal" event={"ID":"10e1249619055b5b051291a965f220aa","Type":"ContainerStarted","Data":"a6117349fba9037ea92ac1e796aaf59e885feb2af886483c76371d88b45ad138"}
Sep 18 19:14:51.095409 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.095398    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:51.097972 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.097564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal" event={"ID":"c3599f1063f207d0364594c6f19dc0d1","Type":"ContainerStarted","Data":"43c338334f3120daf2a30401163afc77675e26ca9fefc8116b01a418abc295b8"}
Sep 18 19:14:51.099344 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.099230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerStarted","Data":"83e7a68327e8caae0a173ffd87c26feb5934e6ac96b9231f0050764f75e96aba"}
Sep 18 19:14:51.100883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.100859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5" event={"ID":"9917eb6e-c6ba-494c-b736-c205153c0e1c","Type":"ContainerStarted","Data":"46f1919fc374ba8a34ca3bd035d53a10d91b97c7de752827277789f2ffc9253a"}
Sep 18 19:14:51.102804 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.102761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-znpkc" event={"ID":"e5c57212-561c-4172-9325-8d50d8b10155","Type":"ContainerStarted","Data":"bea8b598dc797b6290eb0ecf3a2d68e76a98d4480c81eadb5877734341f09d1e"}
Sep 18 19:14:51.106343 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106261    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:14:51.106812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106785    2582 generic.go:356] "Generic (PLEG): container finished" podID="45c34a65-61c3-418d-8599-87cb5547cba6" containerID="4e8ee757aef690e976eaf6bb5368d29d26f6b64bef5ae33f8d9d5124a3f2a147" exitCode=1
Sep 18 19:14:51.106901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"4ea0b2202c27acc49130e9525330ceb6fc779894346310bf6cad6b157abb483e"}
Sep 18 19:14:51.106951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"32fe5adc746104f540fa05a40d2d1df6ddd0c3798692f85a846a121cda4d63b4"}
Sep 18 19:14:51.106951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"ef421e90efdc3afebb8d9f4b91bc8042e52fb22bc27b0be208083ae8cdc0316d"}
Sep 18 19:14:51.107012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.106972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"f9d3d2737a413363a6b91121387d18cd4f8e0b0a29288c605e528af186b2e821"}
Sep 18 19:14:51.107012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.107000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerDied","Data":"4e8ee757aef690e976eaf6bb5368d29d26f6b64bef5ae33f8d9d5124a3f2a147"}
Sep 18 19:14:51.107104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.107030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"5505fb5ff50ba8beb25107275ca25fc4a86ee4650260a516877c333433693adf"}
Sep 18 19:14:51.108243 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.108224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwg8m" event={"ID":"6cc65196-8ae5-48cb-8e49-9e8847db616a","Type":"ContainerStarted","Data":"aa03b99ffa6a5ea9727962c0948c4ed35c7488cc1e87c778041a37adf2bc99dd"}
Sep 18 19:14:51.120147 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.120128    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:51.120363 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.120324    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kbjz4" podStartSLOduration=2.120042196 podStartE2EDuration="21.120308151s" totalImagesPullingTime="19.000265955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.119806576 +0000 UTC m=+23.356933563" watchObservedRunningTime="2026-09-18 19:14:51.120308151 +0000 UTC m=+23.357435141"
Sep 18 19:14:51.120953 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.120937    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"]
Sep 18 19:14:51.179942 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.179899    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwg8m" podStartSLOduration=3.134757659 podStartE2EDuration="22.179884245s" totalImagesPullingTime="19.045126586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.145048142 +0000 UTC m=+23.382175132" watchObservedRunningTime="2026-09-18 19:14:51.179884245 +0000 UTC m=+23.417011293"
Sep 18 19:14:51.221620 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.221567    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mdxcq" podStartSLOduration=2.150464914 podStartE2EDuration="21.221545814s" totalImagesPullingTime="19.0710809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.183029681 +0000 UTC m=+23.420156667" watchObservedRunningTime="2026-09-18 19:14:51.221545814 +0000 UTC m=+23.458672801"
Sep 18 19:14:51.253755 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.252820    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-znpkc" podStartSLOduration=6.023600034 podStartE2EDuration="21.252798255s" totalImagesPullingTime="15.229198221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.252427172 +0000 UTC m=+23.489554159" watchObservedRunningTime="2026-09-18 19:14:51.252798255 +0000 UTC m=+23.489925243"
Sep 18 19:14:51.331027 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:51.330990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7ts68" podStartSLOduration=1.525614727 podStartE2EDuration="21.330976303s" totalImagesPullingTime="19.805361576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.294792973 +0000 UTC m=+23.531919960" watchObservedRunningTime="2026-09-18 19:14:51.330976303 +0000 UTC m=+23.568103288"
Sep 18 19:14:52.111474 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.111204    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="83e7a68327e8caae0a173ffd87c26feb5934e6ac96b9231f0050764f75e96aba" exitCode=0
Sep 18 19:14:52.112224 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.111284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"83e7a68327e8caae0a173ffd87c26feb5934e6ac96b9231f0050764f75e96aba"}
Sep 18 19:14:52.112895 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.112872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5r4z" event={"ID":"8d0b7331-c1a1-4c7f-a9ce-5fd466776677","Type":"ContainerStarted","Data":"3771be24c48c0fd54da3831abba29fd1e0c026dbd9a165726c4b726f3de95425"}
Sep 18 19:14:52.114441 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.114421    2582 generic.go:356] "Generic (PLEG): container finished" podID="c3599f1063f207d0364594c6f19dc0d1" containerID="43c338334f3120daf2a30401163afc77675e26ca9fefc8116b01a418abc295b8" exitCode=0
Sep 18 19:14:52.114550 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.114481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal" event={"ID":"c3599f1063f207d0364594c6f19dc0d1","Type":"ContainerDied","Data":"43c338334f3120daf2a30401163afc77675e26ca9fefc8116b01a418abc295b8"}
Sep 18 19:14:52.114550 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.114510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal" event={"ID":"c3599f1063f207d0364594c6f19dc0d1","Type":"ContainerStarted","Data":"a1c58e707861d87b54bf25453d278a8d92adde73f54a12bace6130edc801da65"}
Sep 18 19:14:52.114919 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.114904    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:52.138742 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.138714    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:52.138817 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:52.138767    2582 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-75.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal"
Sep 18 19:14:52.199807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.199753    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-75.ec2.internal" podStartSLOduration=1.19971722 podStartE2EDuration="1.19971722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:52.144258289 +0000 UTC m=+24.381385275" watchObservedRunningTime="2026-09-18 19:14:52.19971722 +0000 UTC m=+24.436844205"
Sep 18 19:14:52.277456 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.277419    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-75.ec2.internal" podStartSLOduration=22.277404814 podStartE2EDuration="22.277404814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:52.229773455 +0000 UTC m=+24.466900437" watchObservedRunningTime="2026-09-18 19:14:52.277404814 +0000 UTC m=+24.514531800"
Sep 18 19:14:52.293515 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.293493    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:52.811145 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.811040    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:52.2935122Z","UUID":"2c5c8838-8cf8-4c7b-8d54-7a3b012a0024","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:52.812678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.812656    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:52.812678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.812690    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:52.889889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.889865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:52.890049 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:52.889980    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:52.890356 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.890334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:52.890498 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:52.890447    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:52.890580 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:52.890503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:52.890580 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:52.890568    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:53.119055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:53.119014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5" event={"ID":"9917eb6e-c6ba-494c-b736-c205153c0e1c","Type":"ContainerStarted","Data":"2feb5206472baf696e75342954296115a3a2adf14c72a5cde501d4c69f840408"}
Sep 18 19:14:53.130608 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:53.130264    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:14:53.131175 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:53.131123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"44c5440a8af457c07ee3a4d714e7ad888e33e82de3fe1b14786ca39bc6f09509"}
Sep 18 19:14:54.136237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.136201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5" event={"ID":"9917eb6e-c6ba-494c-b736-c205153c0e1c","Type":"ContainerStarted","Data":"34e3f67a35bd0240c2d500ebc1d4fe37963a10f366d38b46c5e930fd4a116cf1"}
Sep 18 19:14:54.165650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.165599    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g5r4z" podStartSLOduration=5.153628697 podStartE2EDuration="24.165581715s" totalImagesPullingTime="19.011953018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:52.277515099 +0000 UTC m=+24.514642084" watchObservedRunningTime="2026-09-18 19:14:54.165581715 +0000 UTC m=+26.402708704"
Sep 18 19:14:54.165848 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.165722    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm9m5" podStartSLOduration=2.734455728 podStartE2EDuration="24.16571431s" totalImagesPullingTime="21.431258582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:54.165043463 +0000 UTC m=+26.402170449" watchObservedRunningTime="2026-09-18 19:14:54.16571431 +0000 UTC m=+26.402841296"
Sep 18 19:14:54.889743 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.889688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:54.889926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.889688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:54.889926 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:54.889843    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:54.889926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:54.889848    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:54.890086 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:54.889946    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:54.890086 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:54.890013    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:55.656186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:55.656154    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:55.656955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:55.656935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:14:56.889365 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:56.889186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:56.889842 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:56.889199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:56.889842 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:56.889435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:56.889842 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:56.889497    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:56.889842 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:56.889209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:56.889842 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:56.889561    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:57.144754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.144647    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="2729de9ebc3c16c988ca5d40ffd70446f552bb9be3d9640cfde2026095740268" exitCode=0
Sep 18 19:14:57.144980 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.144754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"2729de9ebc3c16c988ca5d40ffd70446f552bb9be3d9640cfde2026095740268"}
Sep 18 19:14:57.148248 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.148233    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:14:57.148609 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.148588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"b8c87714beddd82b69a80db2121966b005cda8b0d231487f8011bac51c5fbeee"}
Sep 18 19:14:57.148846 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.148819    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:57.148944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.148886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:57.149055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.149043    2582 scope.go:117] "RemoveContainer" containerID="4e8ee757aef690e976eaf6bb5368d29d26f6b64bef5ae33f8d9d5124a3f2a147"
Sep 18 19:14:57.165253 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:57.165237    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:58.153421 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.153234    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="7eb422a67c1fe42b3c162c3d2d45ae8af63a6bf2f0e6c4bd39828efda44c2b36" exitCode=0
Sep 18 19:14:58.153994 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.153285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"7eb422a67c1fe42b3c162c3d2d45ae8af63a6bf2f0e6c4bd39828efda44c2b36"}
Sep 18 19:14:58.157821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.157805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:14:58.158180 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.158158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" event={"ID":"45c34a65-61c3-418d-8599-87cb5547cba6","Type":"ContainerStarted","Data":"6dadd7f407675853f69b891504d81a91700042e123ebc80265fc05c441ad6639"}
Sep 18 19:14:58.158401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.158375    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:58.172695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.172677    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:14:58.233134 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.233054    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5" podStartSLOduration=9.16847181 podStartE2EDuration="28.233034938s" totalImagesPullingTime="19.064563128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:58.22147482 +0000 UTC m=+30.458601806" watchObservedRunningTime="2026-09-18 19:14:58.233034938 +0000 UTC m=+30.470161924"
Sep 18 19:14:58.233351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.233337    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phr2x"]
Sep 18 19:14:58.233461 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.233448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:58.233561 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:58.233536    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:58.236496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.236472    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-spxh5"]
Sep 18 19:14:58.236592 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.236546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:58.236651 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:58.236606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:14:58.239375 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.239354    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2bqwp"]
Sep 18 19:14:58.239489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:58.239476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:58.239600 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:58.239569    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:59.890923 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:59.890890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:14:59.891289 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:59.890966    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:14:59.891289 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:59.891046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:14:59.891289 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:59.891140    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:14:59.891289 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:14:59.891172    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:14:59.891289 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:14:59.891216    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:15:00.165508 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:00.165432    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="29e106cd76047f580d8063c5916a0621e159f0d4a095c33d0c0855875ece8200" exitCode=0
Sep 18 19:15:00.165640 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:00.165520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"29e106cd76047f580d8063c5916a0621e159f0d4a095c33d0c0855875ece8200"}
Sep 18 19:15:00.707679 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:00.707644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:15:00.707870 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:00.707835    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:15:00.708508 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:00.708485    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-znpkc"
Sep 18 19:15:01.889666 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:01.889631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:01.889666 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:01.889646    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:15:01.889666 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:01.889631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:01.890210 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:01.889801    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2bqwp" podUID="b7cc0d1c-5bcb-4042-ae08-3c097f95298c"
Sep 18 19:15:01.890210 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:01.889849    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-spxh5" podUID="2b36dced-2265-4d55-9ff6-e28b8102b04e"
Sep 18 19:15:01.890210 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:01.889931    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phr2x" podUID="69d4d28c-3322-4118-ab9c-dd3affd62fe5"
Sep 18 19:15:02.474822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:02.474661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:15:02.474822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:02.474701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:02.474822 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.474797    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:02.475129 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.474859    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:34.474840646 +0000 UTC m=+66.711967624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:02.475129 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.474797    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:02.475129 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.474956    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret podName:2b36dced-2265-4d55-9ff6-e28b8102b04e nodeName:}" failed. No retries permitted until 2026-09-18 19:15:34.47493636 +0000 UTC m=+66.712063328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret") pod "global-pull-secret-syncer-spxh5" (UID: "2b36dced-2265-4d55-9ff6-e28b8102b04e") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:02.575786 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:02.575755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:02.575934 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.575898    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:15:02.575934 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.575917    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:15:02.576024 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.575936    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bnvx2 for pod openshift-network-diagnostics/network-check-target-phr2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:02.576024 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:02.575990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2 podName:69d4d28c-3322-4118-ab9c-dd3affd62fe5 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:34.575973636 +0000 UTC m=+66.813100605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnvx2" (UniqueName: "kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2") pod "network-check-target-phr2x" (UID: "69d4d28c-3322-4118-ab9c-dd3affd62fe5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:03.630446 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.630420    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-75.ec2.internal" event="NodeReady"
Sep 18 19:15:03.630860 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.630549    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:15:03.686489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.686462    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jg4cv"]
Sep 18 19:15:03.689313 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.689241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.690234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.690215    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6h4jg"]
Sep 18 19:15:03.691553 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.691529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:15:03.691803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.691786    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:15:03.695289 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.692469    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:15:03.696110 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.696091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.698817 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.698799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:15:03.699044 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.699032    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:15:03.699085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.699051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4mxkp\""
Sep 18 19:15:03.700577 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.700549    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:15:03.706431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.706412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg4cv"]
Sep 18 19:15:03.707233 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.707213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6h4jg"]
Sep 18 19:15:03.783943 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.783916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.784112 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.783979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b83dbf-37c5-4f51-8eaa-9c313b54008f-tmp-dir\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.784112 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.784064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b83dbf-37c5-4f51-8eaa-9c313b54008f-config-volume\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.784112 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.784094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-cert\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.784249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.784118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9npnn\" (UniqueName: \"kubernetes.io/projected/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-kube-api-access-9npnn\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.784249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.784165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p866f\" (UniqueName: \"kubernetes.io/projected/39b83dbf-37c5-4f51-8eaa-9c313b54008f-kube-api-access-p866f\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b83dbf-37c5-4f51-8eaa-9c313b54008f-tmp-dir\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b83dbf-37c5-4f51-8eaa-9c313b54008f-config-volume\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-cert\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.885613 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9npnn\" (UniqueName: \"kubernetes.io/projected/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-kube-api-access-9npnn\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.885613 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p866f\" (UniqueName: \"kubernetes.io/projected/39b83dbf-37c5-4f51-8eaa-9c313b54008f-kube-api-access-p866f\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885613 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885613 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:03.885520    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:03.885613 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:03.885584    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:04.385562369 +0000 UTC m=+36.622689367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:03.885952 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b83dbf-37c5-4f51-8eaa-9c313b54008f-tmp-dir\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.885993 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.885970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39b83dbf-37c5-4f51-8eaa-9c313b54008f-config-volume\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:03.889378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.889355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:03.889378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.889373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:15:03.889542 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.889522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:03.890035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.890012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-cert\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.893450 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:03.893536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893450    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:15:03.893536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:03.893651 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893534    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:03.893706 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893665    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:03.893864 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.893830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:15:03.899024 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.898998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9npnn\" (UniqueName: \"kubernetes.io/projected/9ee23005-dadc-47c4-ba3e-70e9eeadc2ec-kube-api-access-9npnn\") pod \"ingress-canary-6h4jg\" (UID: \"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec\") " pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:03.912158 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:03.912139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p866f\" (UniqueName: \"kubernetes.io/projected/39b83dbf-37c5-4f51-8eaa-9c313b54008f-kube-api-access-p866f\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:04.009737 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:04.009690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6h4jg"
Sep 18 19:15:04.389121 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:04.389089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:04.389274 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:04.389213    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.389274 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:04.389264    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:05.389250791 +0000 UTC m=+37.626377759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.704890 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:04.704717    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6h4jg"]
Sep 18 19:15:04.707997 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:04.707972    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee23005_dadc_47c4_ba3e_70e9eeadc2ec.slice/crio-ae1cc6e8f64f3d0f7937e2019d9cd510bb13863e8d2145dfe95b5692a91823c3 WatchSource:0}: Error finding container ae1cc6e8f64f3d0f7937e2019d9cd510bb13863e8d2145dfe95b5692a91823c3: Status 404 returned error can't find the container with id ae1cc6e8f64f3d0f7937e2019d9cd510bb13863e8d2145dfe95b5692a91823c3
Sep 18 19:15:05.179391 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:05.179352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6h4jg" event={"ID":"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec","Type":"ContainerStarted","Data":"ae1cc6e8f64f3d0f7937e2019d9cd510bb13863e8d2145dfe95b5692a91823c3"}
Sep 18 19:15:05.182656 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:05.182625    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="837a58c43f31d47328076c08ed1f4d1a9b4275f08ad22303e8af0282165332e1" exitCode=0
Sep 18 19:15:05.182790 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:05.182670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"837a58c43f31d47328076c08ed1f4d1a9b4275f08ad22303e8af0282165332e1"}
Sep 18 19:15:05.395396 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:05.395358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:05.395507 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:05.395485    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:05.395575 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:05.395556    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:07.395538148 +0000 UTC m=+39.632665126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:06.189146 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:06.189109    2582 generic.go:356] "Generic (PLEG): container finished" podID="41d787f6-5cce-4ebd-8d40-4a4eedd33ca4" containerID="fa03450e897229703e6ef636442edacd98157f3713b4c01010cf47ef95302dfb" exitCode=0
Sep 18 19:15:06.189805 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:06.189191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerDied","Data":"fa03450e897229703e6ef636442edacd98157f3713b4c01010cf47ef95302dfb"}
Sep 18 19:15:07.193980 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:07.193888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6h4jg" event={"ID":"9ee23005-dadc-47c4-ba3e-70e9eeadc2ec","Type":"ContainerStarted","Data":"58966b4fbb95d0fea72165c0a4bd3a8da8a589049a360c73243647911cfc15cf"}
Sep 18 19:15:07.197574 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:07.197550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" event={"ID":"41d787f6-5cce-4ebd-8d40-4a4eedd33ca4","Type":"ContainerStarted","Data":"de87a852b1d9c992916a127c81a025bf3477b79a666f615a75cd1aa1eb6ba5d2"}
Sep 18 19:15:07.267370 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:07.267320    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6h4jg" podStartSLOduration=1.9856140020000002 podStartE2EDuration="4.267304934s" totalImagesPullingTime="2.281690932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:07.228537148 +0000 UTC m=+39.465664133" watchObservedRunningTime="2026-09-18 19:15:07.267304934 +0000 UTC m=+39.504431943"
Sep 18 19:15:07.267633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:07.267595    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xrpn6" podStartSLOduration=7.868123578 podStartE2EDuration="37.267585563s" totalImagesPullingTime="29.399461985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:07.262797276 +0000 UTC m=+39.499924263" watchObservedRunningTime="2026-09-18 19:15:07.267585563 +0000 UTC m=+39.504712551"
Sep 18 19:15:07.413104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:07.413069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:07.413254 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:07.413209    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:07.413308 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:07.413269    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:11.413249023 +0000 UTC m=+43.650375987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:11.440813 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:11.440768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:11.441272 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:11.440915    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:11.441272 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:11.440979    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:19.440963202 +0000 UTC m=+51.678090166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:16.267281 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.267243    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tl8qx"]
Sep 18 19:15:16.305532 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.305503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tl8qx"]
Sep 18 19:15:16.305676 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.305601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.309013 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.308982    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:15:16.309135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.309030    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m5vl6\""
Sep 18 19:15:16.309135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.309069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:15:16.309681 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.309666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:15:16.309996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.309976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:15:16.316595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.316578    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:15:16.474407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c89fe2-c674-42a9-870d-eb8b2d01de2b-serving-cert\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.474531 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-snapshots\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.474531 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.474531 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qn5\" (UniqueName: \"kubernetes.io/projected/70c89fe2-c674-42a9-870d-eb8b2d01de2b-kube-api-access-s8qn5\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.474679 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.474679 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.474574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-tmp\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-tmp\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c89fe2-c674-42a9-870d-eb8b2d01de2b-serving-cert\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575206 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-snapshots\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575206 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575206 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qn5\" (UniqueName: \"kubernetes.io/projected/70c89fe2-c674-42a9-870d-eb8b2d01de2b-kube-api-access-s8qn5\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575206 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575428 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-tmp\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575690 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/70c89fe2-c674-42a9-870d-eb8b2d01de2b-snapshots\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.575900 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.575880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.576056 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.576043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c89fe2-c674-42a9-870d-eb8b2d01de2b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.578392 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.578376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70c89fe2-c674-42a9-870d-eb8b2d01de2b-serving-cert\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.599286 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.599264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qn5\" (UniqueName: \"kubernetes.io/projected/70c89fe2-c674-42a9-870d-eb8b2d01de2b-kube-api-access-s8qn5\") pod \"insights-operator-6f9c644dd9-tl8qx\" (UID: \"70c89fe2-c674-42a9-870d-eb8b2d01de2b\") " pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.615082 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.615066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx"
Sep 18 19:15:16.905812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:16.905680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tl8qx"]
Sep 18 19:15:16.909944 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:16.909923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c89fe2_c674_42a9_870d_eb8b2d01de2b.slice/crio-1c140ce0f95fa74dd5274698f9da1c2678b0e52b46c7cf0c32a5fc7cc8b7b290 WatchSource:0}: Error finding container 1c140ce0f95fa74dd5274698f9da1c2678b0e52b46c7cf0c32a5fc7cc8b7b290: Status 404 returned error can't find the container with id 1c140ce0f95fa74dd5274698f9da1c2678b0e52b46c7cf0c32a5fc7cc8b7b290
Sep 18 19:15:17.221343 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:17.221306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx" event={"ID":"70c89fe2-c674-42a9-870d-eb8b2d01de2b","Type":"ContainerStarted","Data":"1c140ce0f95fa74dd5274698f9da1c2678b0e52b46c7cf0c32a5fc7cc8b7b290"}
Sep 18 19:15:19.496803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:19.496768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:19.497161 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:19.496893    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:19.497161 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:19.496946    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.496931331 +0000 UTC m=+67.734058294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:20.229288 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.229251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx" event={"ID":"70c89fe2-c674-42a9-870d-eb8b2d01de2b","Type":"ContainerStarted","Data":"edeb5fee1b9529c8c644ab323d96eb9fef21f26c26f879fea97b3683995f7f3e"}
Sep 18 19:15:20.282645 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.282585    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx" podStartSLOduration=1.9039774189999998 podStartE2EDuration="4.282566142s" totalImagesPullingTime="2.378588723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:20.263560663 +0000 UTC m=+52.500687649" watchObservedRunningTime="2026-09-18 19:15:20.282566142 +0000 UTC m=+52.519693127"
Sep 18 19:15:20.283539 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.283518    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xz2d"]
Sep 18 19:15:20.305797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.305763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xz2d"]
Sep 18 19:15:20.305947 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.305843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.308431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.308397    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:15:20.308545 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.308436    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:15:20.308545 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.308481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2glk\""
Sep 18 19:15:20.404254 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.404227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbspl\" (UniqueName: \"kubernetes.io/projected/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-api-access-rbspl\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.404254 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.404254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.404484 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.404278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0437e355-15cb-43cd-b240-cbf87cd7db60-crio-socket\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.404484 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.404320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0437e355-15cb-43cd-b240-cbf87cd7db60-data-volume\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.404484 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.404446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.504842 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.504775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0437e355-15cb-43cd-b240-cbf87cd7db60-crio-socket\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.504842 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.504804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0437e355-15cb-43cd-b240-cbf87cd7db60-data-volume\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.504861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.504943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbspl\" (UniqueName: \"kubernetes.io/projected/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-api-access-rbspl\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.504971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:20.505013    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:20.505055    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.505083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0437e355-15cb-43cd-b240-cbf87cd7db60-crio-socket\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:20.505090    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.005069169 +0000 UTC m=+53.242196135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:20.505127    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:21.005114009 +0000 UTC m=+53.242240973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:20.505219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.505129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0437e355-15cb-43cd-b240-cbf87cd7db60-data-volume\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:20.516541 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:20.516515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbspl\" (UniqueName: \"kubernetes.io/projected/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-api-access-rbspl\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:21.010509 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:21.010477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:21.010661 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:21.010530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:21.010661 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:21.010611    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.010661 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:21.010641    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:21.010779 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:21.010674    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:22.010657642 +0000 UTC m=+54.247784606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:21.010779 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:21.010688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:22.010681771 +0000 UTC m=+54.247808735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:22.018099 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.018066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:22.018469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.018119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:22.018469 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.018200    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:22.018469 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.018256    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.018241345 +0000 UTC m=+56.255368330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:22.018469 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.018204    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:22.018469 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.018321    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.018309643 +0000 UTC m=+56.255436607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:22.175611 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.175587    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kbjz4_9adc468a-d53a-4092-ab1f-73139aea1307/dns-node-resolver/0.log"
Sep 18 19:15:22.192781 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.192760    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"]
Sep 18 19:15:22.194695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.194681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.196824 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.196806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:15:22.196992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.196977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:22.197219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.197203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:22.197769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.197751    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hnjk9\""
Sep 18 19:15:22.207459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.207442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"]
Sep 18 19:15:22.319910 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.319838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88ds\" (UniqueName: \"kubernetes.io/projected/0a8016bd-5820-4b1b-bb81-257bca84905b-kube-api-access-k88ds\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.320031 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.319912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.420322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.420292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88ds\" (UniqueName: \"kubernetes.io/projected/0a8016bd-5820-4b1b-bb81-257bca84905b-kube-api-access-k88ds\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.420439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.420374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.420488 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.420473    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:15:22.420531 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.420522    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls podName:0a8016bd-5820-4b1b-bb81-257bca84905b nodeName:}" failed. No retries permitted until 2026-09-18 19:15:22.920509687 +0000 UTC m=+55.157636655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dhjwn" (UID: "0a8016bd-5820-4b1b-bb81-257bca84905b") : secret "samples-operator-tls" not found
Sep 18 19:15:22.429944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.429917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88ds\" (UniqueName: \"kubernetes.io/projected/0a8016bd-5820-4b1b-bb81-257bca84905b-kube-api-access-k88ds\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.925382 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.925340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:22.925533 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.925484    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:15:22.925568 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:22.925546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls podName:0a8016bd-5820-4b1b-bb81-257bca84905b nodeName:}" failed. No retries permitted until 2026-09-18 19:15:23.925530105 +0000 UTC m=+56.162657091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dhjwn" (UID: "0a8016bd-5820-4b1b-bb81-257bca84905b") : secret "samples-operator-tls" not found
Sep 18 19:15:22.976118 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:22.976098    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwg8m_6cc65196-8ae5-48cb-8e49-9e8847db616a/node-ca/0.log"
Sep 18 19:15:23.025652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.025625    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4slbl"]
Sep 18 19:15:23.027614 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.027596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.030275 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.030259    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8nqgx\""
Sep 18 19:15:23.030527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.030510    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 19:15:23.030972 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.030958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:23.031083 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.031066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:23.031421 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.031407    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 19:15:23.036558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.036538    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 19:15:23.041431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.041412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4slbl"]
Sep 18 19:15:23.126557 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.126530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f844729d-bf68-4b6a-9232-548cea591d4d-serving-cert\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.126557 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.126559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-config\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.126769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.126579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-trusted-ca\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.126769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.126659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzc8\" (UniqueName: \"kubernetes.io/projected/f844729d-bf68-4b6a-9232-548cea591d4d-kube-api-access-rrzc8\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.227587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.227561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f844729d-bf68-4b6a-9232-548cea591d4d-serving-cert\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.227709 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.227599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-config\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.227709 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.227629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-trusted-ca\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.227709 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.227681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzc8\" (UniqueName: \"kubernetes.io/projected/f844729d-bf68-4b6a-9232-548cea591d4d-kube-api-access-rrzc8\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.228270 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.228247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-config\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.228979 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.228957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f844729d-bf68-4b6a-9232-548cea591d4d-trusted-ca\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.230021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.230005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f844729d-bf68-4b6a-9232-548cea591d4d-serving-cert\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.237185 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.237167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzc8\" (UniqueName: \"kubernetes.io/projected/f844729d-bf68-4b6a-9232-548cea591d4d-kube-api-access-rrzc8\") pod \"console-operator-fd4bb46c-4slbl\" (UID: \"f844729d-bf68-4b6a-9232-548cea591d4d\") " pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.337464 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.337428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:23.580441 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.580370    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6h4jg_9ee23005-dadc-47c4-ba3e-70e9eeadc2ec/serve-healthcheck-canary/0.log"
Sep 18 19:15:23.614788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.614758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4slbl"]
Sep 18 19:15:23.619596 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:23.619574    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf844729d_bf68_4b6a_9232_548cea591d4d.slice/crio-6331fcaa20f28a2d092f06f639f4e2f05bd8f6f63ee8a3abd792e4ccf490326f WatchSource:0}: Error finding container 6331fcaa20f28a2d092f06f639f4e2f05bd8f6f63ee8a3abd792e4ccf490326f: Status 404 returned error can't find the container with id 6331fcaa20f28a2d092f06f639f4e2f05bd8f6f63ee8a3abd792e4ccf490326f
Sep 18 19:15:23.932776 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:23.932684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:23.932902 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:23.932825    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:15:23.932902 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:23.932898    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls podName:0a8016bd-5820-4b1b-bb81-257bca84905b nodeName:}" failed. No retries permitted until 2026-09-18 19:15:25.932882441 +0000 UTC m=+58.170009404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dhjwn" (UID: "0a8016bd-5820-4b1b-bb81-257bca84905b") : secret "samples-operator-tls" not found
Sep 18 19:15:24.034036 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:24.034002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:24.034489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:24.034061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:24.034489 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:24.034140    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:24.034489 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:24.034149    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:24.034489 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:24.034209    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.034196045 +0000 UTC m=+60.271323022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:24.034489 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:24.034222    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:28.034214774 +0000 UTC m=+60.271341738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:24.239702 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:24.239666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" event={"ID":"f844729d-bf68-4b6a-9232-548cea591d4d","Type":"ContainerStarted","Data":"6331fcaa20f28a2d092f06f639f4e2f05bd8f6f63ee8a3abd792e4ccf490326f"}
Sep 18 19:15:25.867790 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.867702    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"]
Sep 18 19:15:25.869666 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.869646    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:25.875032 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.874891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 19:15:25.875032 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.874926    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 19:15:25.875032 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.874956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:25.875250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.875066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:25.875922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.875902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9p4mf\""
Sep 18 19:15:25.889011 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.888976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"]
Sep 18 19:15:25.949602 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.949573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13fd9aa-1e69-4055-96ad-bc45f178e04d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:25.949717 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.949657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:25.949717 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.949693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a13fd9aa-1e69-4055-96ad-bc45f178e04d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:25.949717 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:25.949709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6gr\" (UniqueName: \"kubernetes.io/projected/a13fd9aa-1e69-4055-96ad-bc45f178e04d-kube-api-access-jg6gr\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:25.949873 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:25.949801    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:15:25.949873 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:25.949854    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls podName:0a8016bd-5820-4b1b-bb81-257bca84905b nodeName:}" failed. No retries permitted until 2026-09-18 19:15:29.949839115 +0000 UTC m=+62.186966080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dhjwn" (UID: "0a8016bd-5820-4b1b-bb81-257bca84905b") : secret "samples-operator-tls" not found
Sep 18 19:15:26.050033 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.050012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a13fd9aa-1e69-4055-96ad-bc45f178e04d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.050125 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.050038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6gr\" (UniqueName: \"kubernetes.io/projected/a13fd9aa-1e69-4055-96ad-bc45f178e04d-kube-api-access-jg6gr\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.050125 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.050061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13fd9aa-1e69-4055-96ad-bc45f178e04d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.050548 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.050529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a13fd9aa-1e69-4055-96ad-bc45f178e04d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.052351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.052334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a13fd9aa-1e69-4055-96ad-bc45f178e04d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.059572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.059552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6gr\" (UniqueName: \"kubernetes.io/projected/a13fd9aa-1e69-4055-96ad-bc45f178e04d-kube-api-access-jg6gr\") pod \"kube-storage-version-migrator-operator-55889f7b47-7wtwj\" (UID: \"a13fd9aa-1e69-4055-96ad-bc45f178e04d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.075587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.075566    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"]
Sep 18 19:15:26.077779 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.077765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.082862 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.082845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 19:15:26.082958 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.082947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 19:15:26.083048 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.083025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-krsjk\""
Sep 18 19:15:26.083166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.083108    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:15:26.083393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.083377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 19:15:26.108182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.108154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"]
Sep 18 19:15:26.150803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.150752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-config\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.150803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.150780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wkv\" (UniqueName: \"kubernetes.io/projected/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-kube-api-access-l5wkv\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.150930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.150870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.177930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.177908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"
Sep 18 19:15:26.246479 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.246457    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/0.log"
Sep 18 19:15:26.246582 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.246500    2582 generic.go:356] "Generic (PLEG): container finished" podID="f844729d-bf68-4b6a-9232-548cea591d4d" containerID="8522c90bf837aed76bb85f59cb266524c20af6639ce19e0b2975924091ce1d7f" exitCode=255
Sep 18 19:15:26.246582 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.246547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" event={"ID":"f844729d-bf68-4b6a-9232-548cea591d4d","Type":"ContainerDied","Data":"8522c90bf837aed76bb85f59cb266524c20af6639ce19e0b2975924091ce1d7f"}
Sep 18 19:15:26.246790 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.246778    2582 scope.go:117] "RemoveContainer" containerID="8522c90bf837aed76bb85f59cb266524c20af6639ce19e0b2975924091ce1d7f"
Sep 18 19:15:26.251440 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.251421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-config\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.251513 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.251468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wkv\" (UniqueName: \"kubernetes.io/projected/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-kube-api-access-l5wkv\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.251906 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.251881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.252060 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.252041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-config\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.253963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.253942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-serving-cert\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.260229 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.260210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wkv\" (UniqueName: \"kubernetes.io/projected/dc9d8357-12fd-4c1c-b14e-e722dbc83f67-kube-api-access-l5wkv\") pod \"service-ca-operator-57cdc4b768-5dqq2\" (UID: \"dc9d8357-12fd-4c1c-b14e-e722dbc83f67\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.386358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.386326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"
Sep 18 19:15:26.446874 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.446849    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj"]
Sep 18 19:15:26.449906 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:26.449882    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13fd9aa_1e69_4055_96ad_bc45f178e04d.slice/crio-d0369f06da832f8f99e6d945c522f46a351f6882b1e5ab3f323b9521990c9bb6 WatchSource:0}: Error finding container d0369f06da832f8f99e6d945c522f46a351f6882b1e5ab3f323b9521990c9bb6: Status 404 returned error can't find the container with id d0369f06da832f8f99e6d945c522f46a351f6882b1e5ab3f323b9521990c9bb6
Sep 18 19:15:26.662371 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:26.662337    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2"]
Sep 18 19:15:26.666593 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:26.666565    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9d8357_12fd_4c1c_b14e_e722dbc83f67.slice/crio-37026b1f2446c21c6ce60d4f0b47440fc45fe22cf751e6d228e3615f9e2269fa WatchSource:0}: Error finding container 37026b1f2446c21c6ce60d4f0b47440fc45fe22cf751e6d228e3615f9e2269fa: Status 404 returned error can't find the container with id 37026b1f2446c21c6ce60d4f0b47440fc45fe22cf751e6d228e3615f9e2269fa
Sep 18 19:15:27.251792 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.251767    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/1.log"
Sep 18 19:15:27.252249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.252210    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/0.log"
Sep 18 19:15:27.252291 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.252243    2582 generic.go:356] "Generic (PLEG): container finished" podID="f844729d-bf68-4b6a-9232-548cea591d4d" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89" exitCode=255
Sep 18 19:15:27.252340 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.252314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" event={"ID":"f844729d-bf68-4b6a-9232-548cea591d4d","Type":"ContainerDied","Data":"e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"}
Sep 18 19:15:27.252448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.252365    2582 scope.go:117] "RemoveContainer" containerID="8522c90bf837aed76bb85f59cb266524c20af6639ce19e0b2975924091ce1d7f"
Sep 18 19:15:27.252840 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.252781    2582 scope.go:117] "RemoveContainer" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"
Sep 18 19:15:27.253003 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:27.252975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:15:27.254143 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.254119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2" event={"ID":"dc9d8357-12fd-4c1c-b14e-e722dbc83f67","Type":"ContainerStarted","Data":"37026b1f2446c21c6ce60d4f0b47440fc45fe22cf751e6d228e3615f9e2269fa"}
Sep 18 19:15:27.255247 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:27.255224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj" event={"ID":"a13fd9aa-1e69-4055-96ad-bc45f178e04d","Type":"ContainerStarted","Data":"d0369f06da832f8f99e6d945c522f46a351f6882b1e5ab3f323b9521990c9bb6"}
Sep 18 19:15:28.066149 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:28.066109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:28.066339 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:28.066191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:28.066339 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:28.066237    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:15:28.066339 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:28.066319    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.066298006 +0000 UTC m=+68.303424973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : configmap "kube-rbac-proxy" not found
Sep 18 19:15:28.066502 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:28.066364    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:28.066502 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:28.066431    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:36.066410454 +0000 UTC m=+68.303537432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:28.259223 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:28.259200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/1.log"
Sep 18 19:15:28.259599 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:28.259518    2582 scope.go:117] "RemoveContainer" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"
Sep 18 19:15:28.259677 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:28.259658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:15:29.264457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:29.264421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj" event={"ID":"a13fd9aa-1e69-4055-96ad-bc45f178e04d","Type":"ContainerStarted","Data":"d28bbbc503a9ca58957948a15accbbbb14a2f96b258b94706473644013bd52a5"}
Sep 18 19:15:29.265954 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:29.265929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2" event={"ID":"dc9d8357-12fd-4c1c-b14e-e722dbc83f67","Type":"ContainerStarted","Data":"c1a1c2d479b11494825ca6816dbc3e4cd095ece4649e76f00287624c184895af"}
Sep 18 19:15:29.321823 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:29.321780    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj" podStartSLOduration=2.028040731 podStartE2EDuration="4.321764136s" totalImagesPullingTime="2.293723405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:29.288020266 +0000 UTC m=+61.525147252" watchObservedRunningTime="2026-09-18 19:15:29.321764136 +0000 UTC m=+61.558891242"
Sep 18 19:15:29.322089 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:29.322064    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2" podStartSLOduration=1.243387611 podStartE2EDuration="3.322054428s" totalImagesPullingTime="2.078666817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:29.320572337 +0000 UTC m=+61.557699323" watchObservedRunningTime="2026-09-18 19:15:29.322054428 +0000 UTC m=+61.559181411"
Sep 18 19:15:29.983470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:29.983425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:29.983644 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:29.983594    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:15:29.983687 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:29.983670    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls podName:0a8016bd-5820-4b1b-bb81-257bca84905b nodeName:}" failed. No retries permitted until 2026-09-18 19:15:37.983649951 +0000 UTC m=+70.220776924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-dhjwn" (UID: "0a8016bd-5820-4b1b-bb81-257bca84905b") : secret "samples-operator-tls" not found
Sep 18 19:15:30.175997 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:30.175970    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wlzg5"
Sep 18 19:15:32.587950 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.587916    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k9vpm"]
Sep 18 19:15:32.590890 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.590866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.593927 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.593898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dlkgw\""
Sep 18 19:15:32.594036 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.593931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:15:32.594103 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.594069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:15:32.594405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.594389    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:15:32.594405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.594397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:15:32.603566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.603545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k9vpm"]
Sep 18 19:15:32.705936 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.705897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6d2\" (UniqueName: \"kubernetes.io/projected/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-kube-api-access-7m6d2\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.706049 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.705996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-cabundle\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.706049 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.706028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-key\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.806852 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.806827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6d2\" (UniqueName: \"kubernetes.io/projected/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-kube-api-access-7m6d2\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.806981 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.806884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-cabundle\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.806981 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.806917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-key\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.807506 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.807488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-cabundle\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.809352 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.809334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-signing-key\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.820285 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.820261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6d2\" (UniqueName: \"kubernetes.io/projected/cebf8b70-69ef-4d7e-a9a8-156b3f922d72-kube-api-access-7m6d2\") pod \"service-ca-55f6fc7b86-k9vpm\" (UID: \"cebf8b70-69ef-4d7e-a9a8-156b3f922d72\") " pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:32.899408 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:32.899344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm"
Sep 18 19:15:33.172999 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.172932    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-k9vpm"]
Sep 18 19:15:33.176927 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:33.176903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebf8b70_69ef_4d7e_a9a8_156b3f922d72.slice/crio-569ba69833ddf12c781cf0c88b65ab434397c53b01e75a5cc8ce834328b014c7 WatchSource:0}: Error finding container 569ba69833ddf12c781cf0c88b65ab434397c53b01e75a5cc8ce834328b014c7: Status 404 returned error can't find the container with id 569ba69833ddf12c781cf0c88b65ab434397c53b01e75a5cc8ce834328b014c7
Sep 18 19:15:33.278799 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.278762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm" event={"ID":"cebf8b70-69ef-4d7e-a9a8-156b3f922d72","Type":"ContainerStarted","Data":"43d698b2592fd687e5dcd5f96fff2340c2054587d59b3785f405319de4719fa4"}
Sep 18 19:15:33.278983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.278819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm" event={"ID":"cebf8b70-69ef-4d7e-a9a8-156b3f922d72","Type":"ContainerStarted","Data":"569ba69833ddf12c781cf0c88b65ab434397c53b01e75a5cc8ce834328b014c7"}
Sep 18 19:15:33.300262 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.300215    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-k9vpm" podStartSLOduration=1.300199795 podStartE2EDuration="1.300199795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:33.29892059 +0000 UTC m=+65.536047557" watchObservedRunningTime="2026-09-18 19:15:33.300199795 +0000 UTC m=+65.537326781"
Sep 18 19:15:33.337908 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.337882    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:33.338005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.337923    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:33.338246 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:33.338229    2582 scope.go:117] "RemoveContainer" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"
Sep 18 19:15:33.338400 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:33.338382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:15:34.520932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.520902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:15:34.520932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.520936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:34.524962 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.524944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:34.525019 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.524972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:34.532019 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:34.531999    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:15:34.532150 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:34.532067    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs podName:b7cc0d1c-5bcb-4042-ae08-3c097f95298c nodeName:}" failed. No retries permitted until 2026-09-18 19:16:38.532048587 +0000 UTC m=+130.769175565 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs") pod "network-metrics-daemon-2bqwp" (UID: "b7cc0d1c-5bcb-4042-ae08-3c097f95298c") : secret "metrics-daemon-secret" not found
Sep 18 19:15:34.534453 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.534433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b36dced-2265-4d55-9ff6-e28b8102b04e-original-pull-secret\") pod \"global-pull-secret-syncer-spxh5\" (UID: \"2b36dced-2265-4d55-9ff6-e28b8102b04e\") " pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:34.621872 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.621832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:34.625951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.625931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:34.635663 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.635640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:34.645781 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.645760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvx2\" (UniqueName: \"kubernetes.io/projected/69d4d28c-3322-4118-ab9c-dd3affd62fe5-kube-api-access-bnvx2\") pod \"network-check-target-phr2x\" (UID: \"69d4d28c-3322-4118-ab9c-dd3affd62fe5\") " pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:34.815301 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.815222    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4flf6\""
Sep 18 19:15:34.819286 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.819262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:34.820579 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:34.820563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-spxh5"
Sep 18 19:15:35.145720 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:35.145637    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phr2x"]
Sep 18 19:15:35.148819 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:35.148794    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d4d28c_3322_4118_ab9c_dd3affd62fe5.slice/crio-0cb311eae37ff5c2f7c95bb87a7f0c4c39f75e031669c9a3070afb26e7ef12f1 WatchSource:0}: Error finding container 0cb311eae37ff5c2f7c95bb87a7f0c4c39f75e031669c9a3070afb26e7ef12f1: Status 404 returned error can't find the container with id 0cb311eae37ff5c2f7c95bb87a7f0c4c39f75e031669c9a3070afb26e7ef12f1
Sep 18 19:15:35.260454 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:35.260429    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-spxh5"]
Sep 18 19:15:35.262832 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:35.262810    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b36dced_2265_4d55_9ff6_e28b8102b04e.slice/crio-2bb447cb2c4466e0a6ac3b7fa39a8e9bff551d81d054fc46ed3e584538bbcafe WatchSource:0}: Error finding container 2bb447cb2c4466e0a6ac3b7fa39a8e9bff551d81d054fc46ed3e584538bbcafe: Status 404 returned error can't find the container with id 2bb447cb2c4466e0a6ac3b7fa39a8e9bff551d81d054fc46ed3e584538bbcafe
Sep 18 19:15:35.285883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:35.285859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-spxh5" event={"ID":"2b36dced-2265-4d55-9ff6-e28b8102b04e","Type":"ContainerStarted","Data":"2bb447cb2c4466e0a6ac3b7fa39a8e9bff551d81d054fc46ed3e584538bbcafe"}
Sep 18 19:15:35.287222 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:35.287170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phr2x" event={"ID":"69d4d28c-3322-4118-ab9c-dd3affd62fe5","Type":"ContainerStarted","Data":"0cb311eae37ff5c2f7c95bb87a7f0c4c39f75e031669c9a3070afb26e7ef12f1"}
Sep 18 19:15:35.532464 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:35.532432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:15:35.532930 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:35.532612    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:35.532930 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:35.532673    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls podName:39b83dbf-37c5-4f51-8eaa-9c313b54008f nodeName:}" failed. No retries permitted until 2026-09-18 19:16:07.532655941 +0000 UTC m=+99.769782908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls") pod "dns-default-jg4cv" (UID: "39b83dbf-37c5-4f51-8eaa-9c313b54008f") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:36.139122 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:36.139088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:36.139368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:36.139188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:36.139368 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:36.139258    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.139368 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:36.139336    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls podName:0437e355-15cb-43cd-b240-cbf87cd7db60 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:52.13931477 +0000 UTC m=+84.376441751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xz2d" (UID: "0437e355-15cb-43cd-b240-cbf87cd7db60") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:15:36.139659 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:36.139640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0437e355-15cb-43cd-b240-cbf87cd7db60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:38.054538 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:38.054496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:38.057424 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:38.057394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a8016bd-5820-4b1b-bb81-257bca84905b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-dhjwn\" (UID: \"0a8016bd-5820-4b1b-bb81-257bca84905b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:38.105837 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:38.105802    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hnjk9\""
Sep 18 19:15:38.114159 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:38.114135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"
Sep 18 19:15:40.044597 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.044565    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn"]
Sep 18 19:15:40.308009 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.307934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-spxh5" event={"ID":"2b36dced-2265-4d55-9ff6-e28b8102b04e","Type":"ContainerStarted","Data":"27c096d12d7bc76b9bb1910e7eda201c9ac56c810f0eecd3489678ac183f127e"}
Sep 18 19:15:40.309204 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.309179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn" event={"ID":"0a8016bd-5820-4b1b-bb81-257bca84905b","Type":"ContainerStarted","Data":"2e2df020ecdc0fc39be0f56b8e04bd29d605afe8190243adb0436e4cb7165838"}
Sep 18 19:15:40.310509 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.310488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phr2x" event={"ID":"69d4d28c-3322-4118-ab9c-dd3affd62fe5","Type":"ContainerStarted","Data":"4f44d0ea2452c2f421bc1b4f3222f70e11f4d557be2b841dc436160e588a6886"}
Sep 18 19:15:40.310629 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.310614    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:15:40.334571 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.334529    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-spxh5" podStartSLOduration=65.824103441 podStartE2EDuration="1m10.334512885s" totalImagesPullingTime="4.510409444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:40.331541645 +0000 UTC m=+72.568668631" watchObservedRunningTime="2026-09-18 19:15:40.334512885 +0000 UTC m=+72.571639876"
Sep 18 19:15:40.353948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:40.353912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-phr2x" podStartSLOduration=65.745210011 podStartE2EDuration="1m10.35389916s" totalImagesPullingTime="4.608689149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:40.352272321 +0000 UTC m=+72.589399307" watchObservedRunningTime="2026-09-18 19:15:40.35389916 +0000 UTC m=+72.591026146"
Sep 18 19:15:42.319431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:42.319394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn" event={"ID":"0a8016bd-5820-4b1b-bb81-257bca84905b","Type":"ContainerStarted","Data":"fb10d7af07da59f7962e50d83ad9cb8338bf48e2bb108208536e4d349e71739f"}
Sep 18 19:15:42.319810 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:42.319455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn" event={"ID":"0a8016bd-5820-4b1b-bb81-257bca84905b","Type":"ContainerStarted","Data":"66776b7e9e217773de5d6fdd84cc4c17b159d80e55aead406af79ea8b56ba297"}
Sep 18 19:15:42.338741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:42.338683    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-dhjwn" podStartSLOduration=18.515530464 podStartE2EDuration="20.338670641s" totalImagesPullingTime="1.823140177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:42.33774978 +0000 UTC m=+74.574876760" watchObservedRunningTime="2026-09-18 19:15:42.338670641 +0000 UTC m=+74.575797627"
Sep 18 19:15:45.890187 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:45.890160    2582 scope.go:117] "RemoveContainer" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"
Sep 18 19:15:46.333459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.333436    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:15:46.333846 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.333830    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/1.log"
Sep 18 19:15:46.333909 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.333867    2582 generic.go:356] "Generic (PLEG): container finished" podID="f844729d-bf68-4b6a-9232-548cea591d4d" containerID="e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268" exitCode=255
Sep 18 19:15:46.333942 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.333930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" event={"ID":"f844729d-bf68-4b6a-9232-548cea591d4d","Type":"ContainerDied","Data":"e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268"}
Sep 18 19:15:46.333997 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.333985    2582 scope.go:117] "RemoveContainer" containerID="e9e68799ecde65d05081da10ec45fc7627822abcc232f4934738b4b2f5b71f89"
Sep 18 19:15:46.334288 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:46.334262    2582 scope.go:117] "RemoveContainer" containerID="e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268"
Sep 18 19:15:46.334477 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:46.334455    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:15:47.338503 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:47.338474    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:15:52.161652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.161605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:52.166617 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.166588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0437e355-15cb-43cd-b240-cbf87cd7db60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xz2d\" (UID: \"0437e355-15cb-43cd-b240-cbf87cd7db60\") " pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:52.419198 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.419119    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d2glk\""
Sep 18 19:15:52.426961 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.426936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xz2d"
Sep 18 19:15:52.710505 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.710429    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xz2d"]
Sep 18 19:15:52.713328 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:52.713302    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0437e355_15cb_43cd_b240_cbf87cd7db60.slice/crio-ab712ac5b40801bd0eb18b01887fe1975d8bbc22457df154202fd593211f86d9 WatchSource:0}: Error finding container ab712ac5b40801bd0eb18b01887fe1975d8bbc22457df154202fd593211f86d9: Status 404 returned error can't find the container with id ab712ac5b40801bd0eb18b01887fe1975d8bbc22457df154202fd593211f86d9
Sep 18 19:15:52.894544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.894513    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"]
Sep 18 19:15:52.898319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.898302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.900960 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.900934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:15:52.901070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.901017    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:15:52.901199 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.901184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:15:52.901199 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.901192    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:15:52.901695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.901682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:15:52.901760 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.901703    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:15:52.903109 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.903094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:15:52.917972 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.917952    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"]
Sep 18 19:15:52.967519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.967610 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.967610 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.967682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4pl\" (UniqueName: \"kubernetes.io/projected/55e54000-ef24-49b4-94e1-bff3e1dc2e12-kube-api-access-bl4pl\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.967715 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:52.967774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:52.967749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e54000-ef24-49b4-94e1-bff3e1dc2e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e54000-ef24-49b4-94e1-bff3e1dc2e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4pl\" (UniqueName: \"kubernetes.io/projected/55e54000-ef24-49b4-94e1-bff3e1dc2e12-kube-api-access-bl4pl\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.068747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.068623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.069351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.069217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e54000-ef24-49b4-94e1-bff3e1dc2e12-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.071217 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.071195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.071328 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.071237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-ca\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.071328 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.071235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.071328 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.071266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55e54000-ef24-49b4-94e1-bff3e1dc2e12-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.091446 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.091420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4pl\" (UniqueName: \"kubernetes.io/projected/55e54000-ef24-49b4-94e1-bff3e1dc2e12-kube-api-access-bl4pl\") pod \"cluster-proxy-proxy-agent-85946d57c5-hrq4j\" (UID: \"55e54000-ef24-49b4-94e1-bff3e1dc2e12\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.221040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.220963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"
Sep 18 19:15:53.338560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.338538    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:53.338628 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.338594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:15:53.339005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.338988    2582 scope.go:117] "RemoveContainer" containerID="e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268"
Sep 18 19:15:53.339265 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:15:53.339239    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:15:53.359315 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.359297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xz2d" event={"ID":"0437e355-15cb-43cd-b240-cbf87cd7db60","Type":"ContainerStarted","Data":"1fa0eacde3d4fecbb87038a171a676a7fe57eaaae27a47e2f18133c13b488a2f"}
Sep 18 19:15:53.359400 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.359340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xz2d" event={"ID":"0437e355-15cb-43cd-b240-cbf87cd7db60","Type":"ContainerStarted","Data":"eb9f907e955680a2ddcc90321121359a938deeaf3e7ea9174da9f28295627c15"}
Sep 18 19:15:53.359400 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.359361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xz2d" event={"ID":"0437e355-15cb-43cd-b240-cbf87cd7db60","Type":"ContainerStarted","Data":"ab712ac5b40801bd0eb18b01887fe1975d8bbc22457df154202fd593211f86d9"}
Sep 18 19:15:53.539004 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:53.538979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j"]
Sep 18 19:15:53.539385 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:15:53.539367    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55e54000_ef24_49b4_94e1_bff3e1dc2e12.slice/crio-23cb7738c868a3e35505d749f96a53e2f459eed01b6f5674a1247f73fb18e256 WatchSource:0}: Error finding container 23cb7738c868a3e35505d749f96a53e2f459eed01b6f5674a1247f73fb18e256: Status 404 returned error can't find the container with id 23cb7738c868a3e35505d749f96a53e2f459eed01b6f5674a1247f73fb18e256
Sep 18 19:15:54.365902 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:54.365861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j" event={"ID":"55e54000-ef24-49b4-94e1-bff3e1dc2e12","Type":"ContainerStarted","Data":"23cb7738c868a3e35505d749f96a53e2f459eed01b6f5674a1247f73fb18e256"}
Sep 18 19:15:56.375523 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:56.375479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xz2d" event={"ID":"0437e355-15cb-43cd-b240-cbf87cd7db60","Type":"ContainerStarted","Data":"4faf7503a4163278a60b273496e386f98a82c1f0450e27fac51cf682f90e923a"}
Sep 18 19:15:56.410628 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:56.410582    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xz2d" podStartSLOduration=33.453758307 podStartE2EDuration="36.41056884s" totalImagesPullingTime="2.956810533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:15:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:56.408107965 +0000 UTC m=+88.645234964" watchObservedRunningTime="2026-09-18 19:15:56.41056884 +0000 UTC m=+88.647695826"
Sep 18 19:15:58.384224 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:58.384191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j" event={"ID":"55e54000-ef24-49b4-94e1-bff3e1dc2e12","Type":"ContainerStarted","Data":"7ec6508bd0709c43d021b798d0110dfb645ae09366ff1a70d3b8f503096dabbc"}
Sep 18 19:15:58.384224 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:58.384231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j" event={"ID":"55e54000-ef24-49b4-94e1-bff3e1dc2e12","Type":"ContainerStarted","Data":"6c1a433988f753d139eded74376f14709f51fad04a3efd781548411643df2ef6"}
Sep 18 19:15:58.384659 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:58.384245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j" event={"ID":"55e54000-ef24-49b4-94e1-bff3e1dc2e12","Type":"ContainerStarted","Data":"b8ccfab73f71c4e2490c5863d1290afc52149d857b119db112ed0584a9d6e76f"}
Sep 18 19:15:58.434029 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:15:58.433973    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85946d57c5-hrq4j" podStartSLOduration=2.324307258 podStartE2EDuration="6.433957522s" totalImagesPullingTime="4.109650264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:58.430535812 +0000 UTC m=+90.667662797" watchObservedRunningTime="2026-09-18 19:15:58.433957522 +0000 UTC m=+90.671084556"
Sep 18 19:16:02.631741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.631697    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7g5s9"]
Sep 18 19:16:02.635317 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.635300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.639212 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:16:02.639350 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639212    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:02.639350 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639321    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:16:02.639473 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:02.639785 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:02.639873 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.639790    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:16:02.640313 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.640297    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-45754\""
Sep 18 19:16:02.742792 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.742745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.742792 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.742797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z6s\" (UniqueName: \"kubernetes.io/projected/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-kube-api-access-72z6s\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743011 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.742845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-wtmp\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743011 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.742906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-metrics-client-ca\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.743023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-root\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.743068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-tls\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.743094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-sys\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.743125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-textfile\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.743255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.743151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.843719 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.843884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z6s\" (UniqueName: \"kubernetes.io/projected/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-kube-api-access-72z6s\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.843884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-wtmp\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.843884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-metrics-client-ca\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.843884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-root\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-tls\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-sys\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-textfile\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-root\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.843970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-wtmp\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.844040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-sys\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844445 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.844406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-textfile\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844668 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.844599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.844822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.844800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-metrics-client-ca\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.846294 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.846268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.846393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.846339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-node-exporter-tls\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.854559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.854540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z6s\" (UniqueName: \"kubernetes.io/projected/d9f386a8-eb6f-4edf-80cf-872a84b54f0c-kube-api-access-72z6s\") pod \"node-exporter-7g5s9\" (UID: \"d9f386a8-eb6f-4edf-80cf-872a84b54f0c\") " pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.946912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:02.946834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7g5s9"
Sep 18 19:16:02.970928 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:16:02.970900    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f386a8_eb6f_4edf_80cf_872a84b54f0c.slice/crio-c08eef2af96241069b7e917146de4228a19269b6e2d5298d8504f195053bfa05 WatchSource:0}: Error finding container c08eef2af96241069b7e917146de4228a19269b6e2d5298d8504f195053bfa05: Status 404 returned error can't find the container with id c08eef2af96241069b7e917146de4228a19269b6e2d5298d8504f195053bfa05
Sep 18 19:16:03.401681 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:03.401645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7g5s9" event={"ID":"d9f386a8-eb6f-4edf-80cf-872a84b54f0c","Type":"ContainerStarted","Data":"c08eef2af96241069b7e917146de4228a19269b6e2d5298d8504f195053bfa05"}
Sep 18 19:16:04.406333 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:04.406240    2582 generic.go:356] "Generic (PLEG): container finished" podID="d9f386a8-eb6f-4edf-80cf-872a84b54f0c" containerID="42da3646b5f4b8a24786e5a79bb1acf6071ca449d79397bbc02ff8573b329f96" exitCode=0
Sep 18 19:16:04.406686 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:04.406324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7g5s9" event={"ID":"d9f386a8-eb6f-4edf-80cf-872a84b54f0c","Type":"ContainerDied","Data":"42da3646b5f4b8a24786e5a79bb1acf6071ca449d79397bbc02ff8573b329f96"}
Sep 18 19:16:05.411622 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:05.411586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7g5s9" event={"ID":"d9f386a8-eb6f-4edf-80cf-872a84b54f0c","Type":"ContainerStarted","Data":"a0e2ba6192daf4f8e47d4acb203dac4fff29a2e90ddc7f9721f168a77a1f739e"}
Sep 18 19:16:05.412115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:05.411636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7g5s9" event={"ID":"d9f386a8-eb6f-4edf-80cf-872a84b54f0c","Type":"ContainerStarted","Data":"ab747898d11e8923e97f9bd651378ecaf85afcc9ea3441c3a9e76970afe02865"}
Sep 18 19:16:05.472355 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:05.472306    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7g5s9" podStartSLOduration=2.386420101 podStartE2EDuration="3.472289987s" totalImagesPullingTime="1.016607657s" totalInitContainerRuntime="69.262229ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:05.464829601 +0000 UTC m=+97.701956587" watchObservedRunningTime="2026-09-18 19:16:05.472289987 +0000 UTC m=+97.709416973"
Sep 18 19:16:05.890242 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:05.890218    2582 scope.go:117] "RemoveContainer" containerID="e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268"
Sep 18 19:16:05.890388 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:16:05.890365    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-4slbl_openshift-console-operator(f844729d-bf68-4b6a-9232-548cea591d4d)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podUID="f844729d-bf68-4b6a-9232-548cea591d4d"
Sep 18 19:16:07.581924 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:07.581884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:16:07.584362 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:07.584340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39b83dbf-37c5-4f51-8eaa-9c313b54008f-metrics-tls\") pod \"dns-default-jg4cv\" (UID: \"39b83dbf-37c5-4f51-8eaa-9c313b54008f\") " pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:16:07.606137 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:07.606111    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zqv6h\""
Sep 18 19:16:07.613408 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:07.613387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:16:07.886862 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:07.886803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg4cv"]
Sep 18 19:16:07.889849 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:16:07.889814    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b83dbf_37c5_4f51_8eaa_9c313b54008f.slice/crio-eea85aebbf3698b14f44c2f20c1111b6fd90ab0c73330ad0b8611d4869a3be39 WatchSource:0}: Error finding container eea85aebbf3698b14f44c2f20c1111b6fd90ab0c73330ad0b8611d4869a3be39: Status 404 returned error can't find the container with id eea85aebbf3698b14f44c2f20c1111b6fd90ab0c73330ad0b8611d4869a3be39
Sep 18 19:16:08.422107 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:08.422063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4cv" event={"ID":"39b83dbf-37c5-4f51-8eaa-9c313b54008f","Type":"ContainerStarted","Data":"eea85aebbf3698b14f44c2f20c1111b6fd90ab0c73330ad0b8611d4869a3be39"}
Sep 18 19:16:10.430267 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:10.430242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4cv" event={"ID":"39b83dbf-37c5-4f51-8eaa-9c313b54008f","Type":"ContainerStarted","Data":"f9d92b92296c225e25dce4e57041db3d4296f13c662f954660086cc1257806f2"}
Sep 18 19:16:11.317544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:11.317511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-phr2x"
Sep 18 19:16:11.435778 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:11.435745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg4cv" event={"ID":"39b83dbf-37c5-4f51-8eaa-9c313b54008f","Type":"ContainerStarted","Data":"6e19264b7ccfeb2e021f3642a4f3cdcdb31f62b969d1be6cb495c2affa094113"}
Sep 18 19:16:11.436124 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:11.435875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:16:11.466182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:11.466134    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jg4cv" podStartSLOduration=66.169334091 podStartE2EDuration="1m8.466118209s" totalImagesPullingTime="2.296784118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:11.465644695 +0000 UTC m=+103.702771682" watchObservedRunningTime="2026-09-18 19:16:11.466118209 +0000 UTC m=+103.703245226"
Sep 18 19:16:19.891647 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:19.891620    2582 scope.go:117] "RemoveContainer" containerID="e6950b2dff4cd1cdb6ae36fb6e8a7e568cb38ee450d5927b92585e15a9a48268"
Sep 18 19:16:20.467560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:20.467534    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:16:20.467744 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:20.467619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" event={"ID":"f844729d-bf68-4b6a-9232-548cea591d4d","Type":"ContainerStarted","Data":"3b10e307ff30fcc24df741016225fb786aa3b7bb2c150a5aff8c2119b26efe72"}
Sep 18 19:16:20.467937 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:20.467918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:16:20.472599 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:20.472580    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl"
Sep 18 19:16:20.496834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:20.496789    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-4slbl" podStartSLOduration=55.518187538 podStartE2EDuration="57.496773916s" totalImagesPullingTime="1.978586378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:20.494716013 +0000 UTC m=+112.731842999" watchObservedRunningTime="2026-09-18 19:16:20.496773916 +0000 UTC m=+112.733900901"
Sep 18 19:16:21.440479 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:21.440452    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jg4cv"
Sep 18 19:16:38.598782 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:38.598746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:16:38.601225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:38.601193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0d1c-5bcb-4042-ae08-3c097f95298c-metrics-certs\") pod \"network-metrics-daemon-2bqwp\" (UID: \"b7cc0d1c-5bcb-4042-ae08-3c097f95298c\") " pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:16:38.727403 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:38.727371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-thgh8\""
Sep 18 19:16:38.735514 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:38.735493    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2bqwp"
Sep 18 19:16:39.026948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:39.026871    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2bqwp"]
Sep 18 19:16:39.027571 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:16:39.027530    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cc0d1c_5bcb_4042_ae08_3c097f95298c.slice/crio-940f6537f3b2b506f920ecda8f654d7aa3715248458c5f5a80049aa5927864c0 WatchSource:0}: Error finding container 940f6537f3b2b506f920ecda8f654d7aa3715248458c5f5a80049aa5927864c0: Status 404 returned error can't find the container with id 940f6537f3b2b506f920ecda8f654d7aa3715248458c5f5a80049aa5927864c0
Sep 18 19:16:39.528878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:39.528837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bqwp" event={"ID":"b7cc0d1c-5bcb-4042-ae08-3c097f95298c","Type":"ContainerStarted","Data":"940f6537f3b2b506f920ecda8f654d7aa3715248458c5f5a80049aa5927864c0"}
Sep 18 19:16:40.533565 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:40.533533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bqwp" event={"ID":"b7cc0d1c-5bcb-4042-ae08-3c097f95298c","Type":"ContainerStarted","Data":"2be5d6fbaeceac697a090d65fd62e4d1adc6d6c075a6f8741a4e3720d8b945ad"}
Sep 18 19:16:40.533974 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:40.533592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2bqwp" event={"ID":"b7cc0d1c-5bcb-4042-ae08-3c097f95298c","Type":"ContainerStarted","Data":"0af8fa60af61310e169f6b5c8094f504fbccea562565f3e1f2a125375342aae5"}
Sep 18 19:16:40.561969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:40.561923    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2bqwp" podStartSLOduration=129.652869179 podStartE2EDuration="2m10.561909382s" totalImagesPullingTime="909.040203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:40.560550951 +0000 UTC m=+132.797677938" watchObservedRunningTime="2026-09-18 19:16:40.561909382 +0000 UTC m=+132.799036387"
Sep 18 19:16:45.551028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.550969    2582 generic.go:356] "Generic (PLEG): container finished" podID="70c89fe2-c674-42a9-870d-eb8b2d01de2b" containerID="edeb5fee1b9529c8c644ab323d96eb9fef21f26c26f879fea97b3683995f7f3e" exitCode=0
Sep 18 19:16:45.551495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.551043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx" event={"ID":"70c89fe2-c674-42a9-870d-eb8b2d01de2b","Type":"ContainerDied","Data":"edeb5fee1b9529c8c644ab323d96eb9fef21f26c26f879fea97b3683995f7f3e"}
Sep 18 19:16:45.551495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.551350    2582 scope.go:117] "RemoveContainer" containerID="edeb5fee1b9529c8c644ab323d96eb9fef21f26c26f879fea97b3683995f7f3e"
Sep 18 19:16:45.552717 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.552696    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc9d8357-12fd-4c1c-b14e-e722dbc83f67" containerID="c1a1c2d479b11494825ca6816dbc3e4cd095ece4649e76f00287624c184895af" exitCode=0
Sep 18 19:16:45.552792 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.552763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2" event={"ID":"dc9d8357-12fd-4c1c-b14e-e722dbc83f67","Type":"ContainerDied","Data":"c1a1c2d479b11494825ca6816dbc3e4cd095ece4649e76f00287624c184895af"}
Sep 18 19:16:45.553040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:45.553025    2582 scope.go:117] "RemoveContainer" containerID="c1a1c2d479b11494825ca6816dbc3e4cd095ece4649e76f00287624c184895af"
Sep 18 19:16:46.558437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:46.558402    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tl8qx" event={"ID":"70c89fe2-c674-42a9-870d-eb8b2d01de2b","Type":"ContainerStarted","Data":"4d8844df02078fac139ad547722195c915cb980258670cda70494358f664972c"}
Sep 18 19:16:46.563169 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:46.563141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-5dqq2" event={"ID":"dc9d8357-12fd-4c1c-b14e-e722dbc83f67","Type":"ContainerStarted","Data":"00b0b41b9a228257832a85a849783168980ab01869af69226591bd0fdc7d6598"}
Sep 18 19:16:48.826285 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:48.826257    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/init-textfile/0.log"
Sep 18 19:16:49.026843 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:49.026816    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/node-exporter/0.log"
Sep 18 19:16:49.224165 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:49.224142    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/kube-rbac-proxy/0.log"
Sep 18 19:16:54.615975 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:54.615941    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:16:54.815716 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:54.815677    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/3.log"
Sep 18 19:16:59.610952 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:59.610861    2582 generic.go:356] "Generic (PLEG): container finished" podID="a13fd9aa-1e69-4055-96ad-bc45f178e04d" containerID="d28bbbc503a9ca58957948a15accbbbb14a2f96b258b94706473644013bd52a5" exitCode=0
Sep 18 19:16:59.611336 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:59.610947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj" event={"ID":"a13fd9aa-1e69-4055-96ad-bc45f178e04d","Type":"ContainerDied","Data":"d28bbbc503a9ca58957948a15accbbbb14a2f96b258b94706473644013bd52a5"}
Sep 18 19:16:59.611336 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:16:59.611236    2582 scope.go:117] "RemoveContainer" containerID="d28bbbc503a9ca58957948a15accbbbb14a2f96b258b94706473644013bd52a5"
Sep 18 19:17:00.620384 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:17:00.620326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-7wtwj" event={"ID":"a13fd9aa-1e69-4055-96ad-bc45f178e04d","Type":"ContainerStarted","Data":"d85c33d8040c239e1002fbb3b460bdee7c728c9d07a51916b439b91b825336ad"}
Sep 18 19:19:29.727081 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:19:29.727047    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:19:29.731654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:19:29.731634    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:19:29.733814 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:19:29.733793    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:19:29.736938 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:19:29.736921    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:19:29.738468 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:19:29.738453    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:21:41.325284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.325213    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"]
Sep 18 19:21:41.328363 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.328346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.331675 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.331649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:21:41.331827 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.331742    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:21:41.332502 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.332486    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qnm4z\""
Sep 18 19:21:41.387965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.387934    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"]
Sep 18 19:21:41.435164 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.435134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.435318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.435168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.435318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.435237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm7h\" (UniqueName: \"kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.536091 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.536052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.536288 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.536096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.536288 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.536150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm7h\" (UniqueName: \"kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.536506 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.536483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.536567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.536514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.545239 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.545195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm7h\" (UniqueName: \"kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.636524 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.636434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:21:41.919873 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.919801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"]
Sep 18 19:21:41.924806 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:21:41.924782    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960b1230_c038_4166_853a_c9d23459404c.slice/crio-19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379 WatchSource:0}: Error finding container 19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379: Status 404 returned error can't find the container with id 19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379
Sep 18 19:21:41.926697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:41.926680    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:21:42.560511 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:42.560480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444" event={"ID":"960b1230-c038-4166-853a-c9d23459404c","Type":"ContainerStarted","Data":"19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379"}
Sep 18 19:21:49.589430 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:49.589395    2582 generic.go:356] "Generic (PLEG): container finished" podID="960b1230-c038-4166-853a-c9d23459404c" containerID="5c7079322045d77f75f0e205bdc7358f2ed20e71e1f51c074d0a403658b92acd" exitCode=0
Sep 18 19:21:49.589825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:49.589458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444" event={"ID":"960b1230-c038-4166-853a-c9d23459404c","Type":"ContainerDied","Data":"5c7079322045d77f75f0e205bdc7358f2ed20e71e1f51c074d0a403658b92acd"}
Sep 18 19:21:52.603080 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:52.603044    2582 generic.go:356] "Generic (PLEG): container finished" podID="960b1230-c038-4166-853a-c9d23459404c" containerID="4c24870a5d4a59e21b329cc837eb33931e2bd9edb3e9bcb97f96b42ce93fa604" exitCode=0
Sep 18 19:21:52.603558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:21:52.603088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444" event={"ID":"960b1230-c038-4166-853a-c9d23459404c","Type":"ContainerDied","Data":"4c24870a5d4a59e21b329cc837eb33931e2bd9edb3e9bcb97f96b42ce93fa604"}
Sep 18 19:22:00.635632 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:00.635590    2582 generic.go:356] "Generic (PLEG): container finished" podID="960b1230-c038-4166-853a-c9d23459404c" containerID="4d0db967721010c84c8ed6c26c52bed82244a6d9244b1e5683354ea61288038d" exitCode=0
Sep 18 19:22:00.636044 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:00.635671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444" event={"ID":"960b1230-c038-4166-853a-c9d23459404c","Type":"ContainerDied","Data":"4d0db967721010c84c8ed6c26c52bed82244a6d9244b1e5683354ea61288038d"}
Sep 18 19:22:01.768003 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.767964    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:22:01.888209 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.888172    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle\") pod \"960b1230-c038-4166-853a-c9d23459404c\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") "
Sep 18 19:22:01.888209 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.888218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntm7h\" (UniqueName: \"kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h\") pod \"960b1230-c038-4166-853a-c9d23459404c\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") "
Sep 18 19:22:01.888479 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.888245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util\") pod \"960b1230-c038-4166-853a-c9d23459404c\" (UID: \"960b1230-c038-4166-853a-c9d23459404c\") "
Sep 18 19:22:01.888911 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.888884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle" (OuterVolumeSpecName: "bundle") pod "960b1230-c038-4166-853a-c9d23459404c" (UID: "960b1230-c038-4166-853a-c9d23459404c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:22:01.890691 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.890633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h" (OuterVolumeSpecName: "kube-api-access-ntm7h") pod "960b1230-c038-4166-853a-c9d23459404c" (UID: "960b1230-c038-4166-853a-c9d23459404c"). InnerVolumeSpecName "kube-api-access-ntm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:22:01.892465 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.892434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util" (OuterVolumeSpecName: "util") pod "960b1230-c038-4166-853a-c9d23459404c" (UID: "960b1230-c038-4166-853a-c9d23459404c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:22:01.989625 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.989589    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:22:01.989625 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.989617    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntm7h\" (UniqueName: \"kubernetes.io/projected/960b1230-c038-4166-853a-c9d23459404c-kube-api-access-ntm7h\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:22:01.989625 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:01.989629    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/960b1230-c038-4166-853a-c9d23459404c-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:22:02.644588 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:02.644551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444" event={"ID":"960b1230-c038-4166-853a-c9d23459404c","Type":"ContainerDied","Data":"19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379"}
Sep 18 19:22:02.644780 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:02.644604    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c4ff20b4843e59b12295cde38a0da326756ee437cacc8461a06052ad54f379"
Sep 18 19:22:02.644780 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:02.644566    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aj9444"
Sep 18 19:22:20.798576 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798543    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"]
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798791    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="util"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798803    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="util"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798818    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="extract"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798824    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="extract"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798833    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="pull"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798839    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="pull"
Sep 18 19:22:20.799055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.798886    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="960b1230-c038-4166-853a-c9d23459404c" containerName="extract"
Sep 18 19:22:20.804359 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.804336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:20.811554 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.811535    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:22:20.812686 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.812669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:22:20.812755 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.812696    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:22:20.812867 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.812849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:22:20.812951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.812934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 19:22:20.813328 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.813315    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lbv7p\""
Sep 18 19:22:20.817436 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.817413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"]
Sep 18 19:22:20.934964 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.934937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:20.935117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.934995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgb9z\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-kube-api-access-jgb9z\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:20.935117 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.935041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7172b923-4939-474f-b31b-01e732bd3924-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:20.992130 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.992099    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-285wk"]
Sep 18 19:22:20.994045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.994029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:20.999282 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:20.999263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:22:21.014040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.014019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-285wk"]
Sep 18 19:22:21.036137 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.036113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgb9z\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-kube-api-access-jgb9z\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.036250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.036164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7172b923-4939-474f-b31b-01e732bd3924-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.036250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.036215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.036599 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.036579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7172b923-4939-474f-b31b-01e732bd3924-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.038837 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.038819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.052208 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.052132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgb9z\" (UniqueName: \"kubernetes.io/projected/7172b923-4939-474f-b31b-01e732bd3924-kube-api-access-jgb9z\") pod \"keda-metrics-apiserver-77857b7b98-jr7pz\" (UID: \"7172b923-4939-474f-b31b-01e732bd3924\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.114823 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.114797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:21.136802 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.136768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8zb\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-kube-api-access-rp8zb\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.136932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.136826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-certificates\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.237708 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.237680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8zb\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-kube-api-access-rp8zb\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.237902 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.237749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-certificates\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.240341 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.240310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-certificates\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.253570 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.253538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8zb\" (UniqueName: \"kubernetes.io/projected/2c6c7eab-6a14-4c65-a481-4c96948cb971-kube-api-access-rp8zb\") pod \"keda-admission-6c4d864b86-285wk\" (UID: \"2c6c7eab-6a14-4c65-a481-4c96948cb971\") " pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.303484 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.303418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:21.409775 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.409744    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"]
Sep 18 19:22:21.412751 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:22:21.412710    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7172b923_4939_474f_b31b_01e732bd3924.slice/crio-1b51fe7be3530034a9acb5acdece9dd9b71ca76051ea4b834a325bb55fbc029a WatchSource:0}: Error finding container 1b51fe7be3530034a9acb5acdece9dd9b71ca76051ea4b834a325bb55fbc029a: Status 404 returned error can't find the container with id 1b51fe7be3530034a9acb5acdece9dd9b71ca76051ea4b834a325bb55fbc029a
Sep 18 19:22:21.580317 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.580230    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-285wk"]
Sep 18 19:22:21.582573 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:22:21.582542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6c7eab_6a14_4c65_a481_4c96948cb971.slice/crio-8fca464b64bf791f7bc21265798ae2b672e49845ecc74f138eb452d43578d36e WatchSource:0}: Error finding container 8fca464b64bf791f7bc21265798ae2b672e49845ecc74f138eb452d43578d36e: Status 404 returned error can't find the container with id 8fca464b64bf791f7bc21265798ae2b672e49845ecc74f138eb452d43578d36e
Sep 18 19:22:21.714180 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.714149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-285wk" event={"ID":"2c6c7eab-6a14-4c65-a481-4c96948cb971","Type":"ContainerStarted","Data":"8fca464b64bf791f7bc21265798ae2b672e49845ecc74f138eb452d43578d36e"}
Sep 18 19:22:21.715225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:21.715205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz" event={"ID":"7172b923-4939-474f-b31b-01e732bd3924","Type":"ContainerStarted","Data":"1b51fe7be3530034a9acb5acdece9dd9b71ca76051ea4b834a325bb55fbc029a"}
Sep 18 19:22:23.724387 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:23.724353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-285wk" event={"ID":"2c6c7eab-6a14-4c65-a481-4c96948cb971","Type":"ContainerStarted","Data":"23133d33339279b44b6a9f46a75c9f08fbefbf7a86cd45fb67a1e87d58a27e3d"}
Sep 18 19:22:23.724799 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:23.724465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:22:27.741207 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:27.741170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz" event={"ID":"7172b923-4939-474f-b31b-01e732bd3924","Type":"ContainerStarted","Data":"cbf5c1692e2c079b26c5ff79e116580c40391fabf1f26a384d821b9c55997a60"}
Sep 18 19:22:27.741664 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:27.741373    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:27.773132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:27.773085    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-285wk" podStartSLOduration=6.006640164 podStartE2EDuration="7.77306594s" totalImagesPullingTime="1.766425776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:23.754455515 +0000 UTC m=+475.991582504" watchObservedRunningTime="2026-09-18 19:22:27.77306594 +0000 UTC m=+480.010192916"
Sep 18 19:22:38.751773 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:38.751723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz"
Sep 18 19:22:38.784972 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:38.784915    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jr7pz" podStartSLOduration=13.176196269 podStartE2EDuration="18.784895145s" totalImagesPullingTime="5.608698876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:27.780485381 +0000 UTC m=+480.017612367" watchObservedRunningTime="2026-09-18 19:22:38.784895145 +0000 UTC m=+491.022022135"
Sep 18 19:22:44.731135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:22:44.731090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-285wk"
Sep 18 19:23:12.704752 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.704648    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"]
Sep 18 19:23:12.707839 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.707822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.715504 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.715488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:23:12.715839 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.715824    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qnm4z\""
Sep 18 19:23:12.716135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.716121    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:23:12.723691 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.723670    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"]
Sep 18 19:23:12.810978 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.810939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.811157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.810998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84nn\" (UniqueName: \"kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.811157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.811084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.911620 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.911588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.911867 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.911638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.911867 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.911687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84nn\" (UniqueName: \"kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.912010 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.911987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.912010 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.912001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:12.922210 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:12.922185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84nn\" (UniqueName: \"kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:13.016373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:13.016339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:13.294377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:13.294356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"]
Sep 18 19:23:13.296693 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:23:13.296672    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0b8178_3ec9_4002_b13c_bda83daedc37.slice/crio-f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302 WatchSource:0}: Error finding container f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302: Status 404 returned error can't find the container with id f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302
Sep 18 19:23:13.918588 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:13.918553    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerID="4d3422dd8b28f3cffb4aefd31123f717c1dfdddd31db388cfdfb51d65e154f44" exitCode=0
Sep 18 19:23:13.919015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:13.918629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt" event={"ID":"cb0b8178-3ec9-4002-b13c-bda83daedc37","Type":"ContainerDied","Data":"4d3422dd8b28f3cffb4aefd31123f717c1dfdddd31db388cfdfb51d65e154f44"}
Sep 18 19:23:13.919015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:13.918666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt" event={"ID":"cb0b8178-3ec9-4002-b13c-bda83daedc37","Type":"ContainerStarted","Data":"f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302"}
Sep 18 19:23:16.932232 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:16.932192    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerID="820eb1d4f29ae7866ea97badc505ceea79c40c1d80e1ccc2fae14d7c9309c232" exitCode=0
Sep 18 19:23:16.932647 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:16.932283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt" event={"ID":"cb0b8178-3ec9-4002-b13c-bda83daedc37","Type":"ContainerDied","Data":"820eb1d4f29ae7866ea97badc505ceea79c40c1d80e1ccc2fae14d7c9309c232"}
Sep 18 19:23:17.937888 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:17.937856    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerID="1ddc0b593e6af75762bf5b4ac36aa896beda65a46bcdd2d481b9bc478dbc4b61" exitCode=0
Sep 18 19:23:17.938250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:17.937947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt" event={"ID":"cb0b8178-3ec9-4002-b13c-bda83daedc37","Type":"ContainerDied","Data":"1ddc0b593e6af75762bf5b4ac36aa896beda65a46bcdd2d481b9bc478dbc4b61"}
Sep 18 19:23:19.067002 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.066978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:19.154364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.154326    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util\") pod \"cb0b8178-3ec9-4002-b13c-bda83daedc37\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") "
Sep 18 19:23:19.154544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.154383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c84nn\" (UniqueName: \"kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn\") pod \"cb0b8178-3ec9-4002-b13c-bda83daedc37\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") "
Sep 18 19:23:19.154544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.154415    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle\") pod \"cb0b8178-3ec9-4002-b13c-bda83daedc37\" (UID: \"cb0b8178-3ec9-4002-b13c-bda83daedc37\") "
Sep 18 19:23:19.155164 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.155142    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle" (OuterVolumeSpecName: "bundle") pod "cb0b8178-3ec9-4002-b13c-bda83daedc37" (UID: "cb0b8178-3ec9-4002-b13c-bda83daedc37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:23:19.156677 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.156650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn" (OuterVolumeSpecName: "kube-api-access-c84nn") pod "cb0b8178-3ec9-4002-b13c-bda83daedc37" (UID: "cb0b8178-3ec9-4002-b13c-bda83daedc37"). InnerVolumeSpecName "kube-api-access-c84nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:23:19.158766 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.158743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util" (OuterVolumeSpecName: "util") pod "cb0b8178-3ec9-4002-b13c-bda83daedc37" (UID: "cb0b8178-3ec9-4002-b13c-bda83daedc37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:23:19.255044 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.255012    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84nn\" (UniqueName: \"kubernetes.io/projected/cb0b8178-3ec9-4002-b13c-bda83daedc37-kube-api-access-c84nn\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:19.255044 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.255043    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:19.255216 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.255053    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb0b8178-3ec9-4002-b13c-bda83daedc37-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:19.947058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.947027    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt"
Sep 18 19:23:19.947237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.947028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bprgt" event={"ID":"cb0b8178-3ec9-4002-b13c-bda83daedc37","Type":"ContainerDied","Data":"f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302"}
Sep 18 19:23:19.947237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:19.947158    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f12c9bba3c4a73c422386b4b801cae5a6c67e3c4439c5fb212c0ec475b5302"
Sep 18 19:23:34.900199 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900162    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"]
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900431    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="extract"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900443    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="extract"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900452    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="util"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900458    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="util"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900464    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="pull"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900469    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="pull"
Sep 18 19:23:34.900569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.900515    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0b8178-3ec9-4002-b13c-bda83daedc37" containerName="extract"
Sep 18 19:23:34.904132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.904116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:34.906878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.906849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qnm4z\""
Sep 18 19:23:34.907001 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.906910    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:23:34.907001 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.906939    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:23:34.916009 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.915987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"]
Sep 18 19:23:34.979442 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.979402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:34.979442 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.979446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:34.979666 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:34.979548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7245\" (UniqueName: \"kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.080242 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.080205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7245\" (UniqueName: \"kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.080419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.080259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.080419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.080279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.080662 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.080640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.080813 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.080678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.089091 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.089069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7245\" (UniqueName: \"kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.213293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.213212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:35.497016 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:35.496985    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"]
Sep 18 19:23:35.501856 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:23:35.501832    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b8d34e_b89f_4348_b9a2_9b3b80bc8a76.slice/crio-c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c WatchSource:0}: Error finding container c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c: Status 404 returned error can't find the container with id c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c
Sep 18 19:23:36.006837 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:36.006799    2582 generic.go:356] "Generic (PLEG): container finished" podID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerID="0fe5ae68a40fc780afccc94132c78a1f1eea0890b06465305480a8e953702a8c" exitCode=0
Sep 18 19:23:36.007235 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:36.006854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9" event={"ID":"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76","Type":"ContainerDied","Data":"0fe5ae68a40fc780afccc94132c78a1f1eea0890b06465305480a8e953702a8c"}
Sep 18 19:23:36.007235 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:36.006906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9" event={"ID":"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76","Type":"ContainerStarted","Data":"c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c"}
Sep 18 19:23:39.020244 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:39.020209    2582 generic.go:356] "Generic (PLEG): container finished" podID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerID="f0a448013ab85cf6da4f218a8b83f2f2b654793e08bdc2359c1eb9510174a16b" exitCode=0
Sep 18 19:23:39.020643 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:39.020296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9" event={"ID":"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76","Type":"ContainerDied","Data":"f0a448013ab85cf6da4f218a8b83f2f2b654793e08bdc2359c1eb9510174a16b"}
Sep 18 19:23:40.026095 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:40.026066    2582 generic.go:356] "Generic (PLEG): container finished" podID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerID="13f45367d3e782202b375ef5c17b74a7dff87b1f262e368369a262b03f69e7ce" exitCode=0
Sep 18 19:23:40.026476 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:40.026109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9" event={"ID":"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76","Type":"ContainerDied","Data":"13f45367d3e782202b375ef5c17b74a7dff87b1f262e368369a262b03f69e7ce"}
Sep 18 19:23:41.152318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.152288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:23:41.225848 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.225812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util\") pod \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") "
Sep 18 19:23:41.226041 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.225896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle\") pod \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") "
Sep 18 19:23:41.226041 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.225936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7245\" (UniqueName: \"kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245\") pod \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\" (UID: \"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76\") "
Sep 18 19:23:41.226273 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.226250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle" (OuterVolumeSpecName: "bundle") pod "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" (UID: "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:23:41.228138 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.228115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245" (OuterVolumeSpecName: "kube-api-access-q7245") pod "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" (UID: "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76"). InnerVolumeSpecName "kube-api-access-q7245". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:23:41.230378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.230345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util" (OuterVolumeSpecName: "util") pod "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" (UID: "58b8d34e-b89f-4348-b9a2-9b3b80bc8a76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:23:41.327080 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.327009    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:41.327080 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.327032    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:41.327080 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:41.327040    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7245\" (UniqueName: \"kubernetes.io/projected/58b8d34e-b89f-4348-b9a2-9b3b80bc8a76-kube-api-access-q7245\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:23:42.035606 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:42.035574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9" event={"ID":"58b8d34e-b89f-4348-b9a2-9b3b80bc8a76","Type":"ContainerDied","Data":"c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c"}
Sep 18 19:23:42.035606 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:42.035621    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21807f135d1aa73714a7a0e0d21df2656fc6d8ed205fc5d44f42d742e59240c"
Sep 18 19:23:42.035836 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:23:42.035621    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdcps9"
Sep 18 19:24:07.503993 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.503956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"]
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504214    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="util"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504225    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="util"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="extract"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="extract"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504254    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="pull"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504259    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="pull"
Sep 18 19:24:07.504439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.504300    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b8d34e-b89f-4348-b9a2-9b3b80bc8a76" containerName="extract"
Sep 18 19:24:07.512142 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.512119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.516421 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.516397    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 19:24:07.516536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.516462    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bd5t7\""
Sep 18 19:24:07.516874 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.516856    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 19:24:07.516973 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.516950    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 19:24:07.517040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.516994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 19:24:07.517101 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.517085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:24:07.529515 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.529495    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"]
Sep 18 19:24:07.626204 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.626170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.626204 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.626204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdf9\" (UniqueName: \"kubernetes.io/projected/8299c0f7-acdc-49ed-ad65-f332d0d899f4-kube-api-access-9xdf9\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.626407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.626233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8299c0f7-acdc-49ed-ad65-f332d0d899f4-manager-config\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.626407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.626319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.727020 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.726984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.727144 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.727029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.727144 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.727048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdf9\" (UniqueName: \"kubernetes.io/projected/8299c0f7-acdc-49ed-ad65-f332d0d899f4-kube-api-access-9xdf9\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.727216 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.727180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8299c0f7-acdc-49ed-ad65-f332d0d899f4-manager-config\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.727816 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.727795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8299c0f7-acdc-49ed-ad65-f332d0d899f4-manager-config\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.729544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.729517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.729646 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.729594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8299c0f7-acdc-49ed-ad65-f332d0d899f4-metrics-cert\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.749513 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.749486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdf9\" (UniqueName: \"kubernetes.io/projected/8299c0f7-acdc-49ed-ad65-f332d0d899f4-kube-api-access-9xdf9\") pod \"lws-controller-manager-85dd87cf5-rfcw8\" (UID: \"8299c0f7-acdc-49ed-ad65-f332d0d899f4\") " pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:07.821336 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:07.821297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:08.107919 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:08.107889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"]
Sep 18 19:24:08.109885 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:08.109855    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8299c0f7_acdc_49ed_ad65_f332d0d899f4.slice/crio-90a04d52e3f23b31ad600d73ecd7a69d30df7044580eb3e3bf39e6487f43403e WatchSource:0}: Error finding container 90a04d52e3f23b31ad600d73ecd7a69d30df7044580eb3e3bf39e6487f43403e: Status 404 returned error can't find the container with id 90a04d52e3f23b31ad600d73ecd7a69d30df7044580eb3e3bf39e6487f43403e
Sep 18 19:24:08.133475 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:08.133448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8" event={"ID":"8299c0f7-acdc-49ed-ad65-f332d0d899f4","Type":"ContainerStarted","Data":"90a04d52e3f23b31ad600d73ecd7a69d30df7044580eb3e3bf39e6487f43403e"}
Sep 18 19:24:10.143254 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:10.143216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8" event={"ID":"8299c0f7-acdc-49ed-ad65-f332d0d899f4","Type":"ContainerStarted","Data":"cc47e86d1317f000ea4af75fb0347a758235093a99f3607298cda7ec83afdeed"}
Sep 18 19:24:10.143674 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:10.143367    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:10.228136 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:10.228079    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8" podStartSLOduration=1.334266577 podStartE2EDuration="3.228060739s" totalImagesPullingTime="1.893794162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:10.227324534 +0000 UTC m=+582.464451522" watchObservedRunningTime="2026-09-18 19:24:10.228060739 +0000 UTC m=+582.465187725"
Sep 18 19:24:21.149884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:21.149847    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85dd87cf5-rfcw8"
Sep 18 19:24:29.759483 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:29.759456    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:24:29.761390 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:29.761367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:24:29.767166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:29.767144    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:24:29.769271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:29.769249    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:24:30.305655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.305623    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"]
Sep 18 19:24:30.308944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.308927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.311515 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.311488    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qnm4z\""
Sep 18 19:24:30.311635 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.311518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:24:30.312073 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.312059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:24:30.319369 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.319347    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"]
Sep 18 19:24:30.390437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.390403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqxl\" (UniqueName: \"kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.390437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.390438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.390634 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.390519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.396773 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.396751    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"]
Sep 18 19:24:30.399937 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.399923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.414609 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.414584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"]
Sep 18 19:24:30.491326 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.491326 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.491575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.491575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqxl\" (UniqueName: \"kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.491575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.491575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlscw\" (UniqueName: \"kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.491830 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.491830 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.491804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.498834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.498810    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"]
Sep 18 19:24:30.502105 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.502090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.512536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.512510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqxl\" (UniqueName: \"kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.516242 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.516222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"]
Sep 18 19:24:30.592226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.592226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mx2c\" (UniqueName: \"kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.592226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlscw\" (UniqueName: \"kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.592427 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.592427 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.592427 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.592563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.592601 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.592577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.606111 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.606085    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"]
Sep 18 19:24:30.607203 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.607178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlscw\" (UniqueName: \"kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.609390 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.609371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.617935 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.617913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:30.625330 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.625310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"]
Sep 18 19:24:30.693695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mx2c\" (UniqueName: \"kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.693847 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmwx\" (UniqueName: \"kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.693847 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.693847 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.693965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.693965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.693922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.694255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.694237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.694292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.694253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.703779 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.703751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mx2c\" (UniqueName: \"kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.708603 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.708577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:30.795019 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.794980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmwx\" (UniqueName: \"kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.795358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.795026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.795358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.795054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.795358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.795335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.795457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.795426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.805151 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.805125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmwx\" (UniqueName: \"kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.810961 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.810939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:30.900278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.900251    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"]
Sep 18 19:24:30.902068 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:30.902043    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11caead_a5ea_4f4b_a1fa_a34ffeb094fa.slice/crio-619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13 WatchSource:0}: Error finding container 619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13: Status 404 returned error can't find the container with id 619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13
Sep 18 19:24:30.920564 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.920540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:30.991850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:30.991819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"]
Sep 18 19:24:30.994842 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:30.994784    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945a9d6c_8a0b_4f78_9855_8fb1bdfbaec3.slice/crio-f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3 WatchSource:0}: Error finding container f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3: Status 404 returned error can't find the container with id f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3
Sep 18 19:24:31.086781 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.086677    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"]
Sep 18 19:24:31.088670 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:31.088643    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1fd185_999c_46a9_ae2c_d26055044aed.slice/crio-0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14 WatchSource:0}: Error finding container 0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14: Status 404 returned error can't find the container with id 0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14
Sep 18 19:24:31.203070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.203041    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"]
Sep 18 19:24:31.227563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.227526    2582 generic.go:356] "Generic (PLEG): container finished" podID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerID="0a0a3d3895678a55a3a4f57f2afa23616aef9b72ab4fca8dbe0e4715e26f884b" exitCode=0
Sep 18 19:24:31.227716 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.227615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm" event={"ID":"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3","Type":"ContainerDied","Data":"0a0a3d3895678a55a3a4f57f2afa23616aef9b72ab4fca8dbe0e4715e26f884b"}
Sep 18 19:24:31.227716 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.227665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm" event={"ID":"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3","Type":"ContainerStarted","Data":"f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3"}
Sep 18 19:24:31.229328 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.229304    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerID="eb6ddae320eca5baf8682d613b09d5ff18145153295dea0b56b85aecd4f50017" exitCode=0
Sep 18 19:24:31.229421 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.229405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg" event={"ID":"2c1fd185-999c-46a9-ae2c-d26055044aed","Type":"ContainerDied","Data":"eb6ddae320eca5baf8682d613b09d5ff18145153295dea0b56b85aecd4f50017"}
Sep 18 19:24:31.229496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.229444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg" event={"ID":"2c1fd185-999c-46a9-ae2c-d26055044aed","Type":"ContainerStarted","Data":"0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14"}
Sep 18 19:24:31.231045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.231025    2582 generic.go:356] "Generic (PLEG): container finished" podID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerID="3004c0f37ea1fb90660b8d7961f83c721fbeca49e2edeb794e8db8f1e07e4690" exitCode=0
Sep 18 19:24:31.231130 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.231071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x" event={"ID":"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa","Type":"ContainerDied","Data":"3004c0f37ea1fb90660b8d7961f83c721fbeca49e2edeb794e8db8f1e07e4690"}
Sep 18 19:24:31.231130 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:31.231097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x" event={"ID":"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa","Type":"ContainerStarted","Data":"619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13"}
Sep 18 19:24:31.234475 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:31.234447    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e09b2cc_f21f_40e7_942c_8af99bb60ec3.slice/crio-02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399 WatchSource:0}: Error finding container 02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399: Status 404 returned error can't find the container with id 02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399
Sep 18 19:24:32.237304 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.237266    2582 generic.go:356] "Generic (PLEG): container finished" podID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerID="f32fe95b1cf78bc7412ac2c6619af68bd3025d91f12b9eff11b140e3f068a88d" exitCode=0
Sep 18 19:24:32.237658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.237324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg" event={"ID":"1e09b2cc-f21f-40e7-942c-8af99bb60ec3","Type":"ContainerDied","Data":"f32fe95b1cf78bc7412ac2c6619af68bd3025d91f12b9eff11b140e3f068a88d"}
Sep 18 19:24:32.237658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.237385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg" event={"ID":"1e09b2cc-f21f-40e7-942c-8af99bb60ec3","Type":"ContainerStarted","Data":"02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399"}
Sep 18 19:24:32.239151 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.239125    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerID="a483758a01d804c8c2b474b98b6999e09f9afcb76afee8ee86a4dd639068f902" exitCode=0
Sep 18 19:24:32.239276 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.239150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg" event={"ID":"2c1fd185-999c-46a9-ae2c-d26055044aed","Type":"ContainerDied","Data":"a483758a01d804c8c2b474b98b6999e09f9afcb76afee8ee86a4dd639068f902"}
Sep 18 19:24:32.241389 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.241370    2582 generic.go:356] "Generic (PLEG): container finished" podID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerID="25266e0fbb1cdb57205d53a673f9e8996f865e313828c4954ce461e4973119ac" exitCode=0
Sep 18 19:24:32.241469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:32.241428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x" event={"ID":"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa","Type":"ContainerDied","Data":"25266e0fbb1cdb57205d53a673f9e8996f865e313828c4954ce461e4973119ac"}
Sep 18 19:24:33.247429 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.247402    2582 generic.go:356] "Generic (PLEG): container finished" podID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerID="b5280bbcd708352c7ac56a5cc54741ed4f36d2986c5e5978dc87873960de73c9" exitCode=0
Sep 18 19:24:33.247845 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.247495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg" event={"ID":"1e09b2cc-f21f-40e7-942c-8af99bb60ec3","Type":"ContainerDied","Data":"b5280bbcd708352c7ac56a5cc54741ed4f36d2986c5e5978dc87873960de73c9"}
Sep 18 19:24:33.249482 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.249461    2582 generic.go:356] "Generic (PLEG): container finished" podID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerID="bd49d233f1daf21931948fbede906e465bad61f93c84fa30eaeca7ff0fec3130" exitCode=0
Sep 18 19:24:33.249569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.249540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm" event={"ID":"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3","Type":"ContainerDied","Data":"bd49d233f1daf21931948fbede906e465bad61f93c84fa30eaeca7ff0fec3130"}
Sep 18 19:24:33.251867 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.251842    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerID="dccba5efa74a95b4f37625e2d89a571a18601698721612fe60abde31e0e2c285" exitCode=0
Sep 18 19:24:33.251963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.251905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg" event={"ID":"2c1fd185-999c-46a9-ae2c-d26055044aed","Type":"ContainerDied","Data":"dccba5efa74a95b4f37625e2d89a571a18601698721612fe60abde31e0e2c285"}
Sep 18 19:24:33.253957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.253937    2582 generic.go:356] "Generic (PLEG): container finished" podID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerID="e411d9bb797c65825578610e79ab982cbe34d77ecf0e0ac485d5644c63e0b171" exitCode=0
Sep 18 19:24:33.254035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:33.254000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x" event={"ID":"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa","Type":"ContainerDied","Data":"e411d9bb797c65825578610e79ab982cbe34d77ecf0e0ac485d5644c63e0b171"}
Sep 18 19:24:34.260576 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.260541    2582 generic.go:356] "Generic (PLEG): container finished" podID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerID="34b5db6899997165077ee526f805d9f6e08b150bc0a57ecaa17c16d494f81c18" exitCode=0
Sep 18 19:24:34.260983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.260616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg" event={"ID":"1e09b2cc-f21f-40e7-942c-8af99bb60ec3","Type":"ContainerDied","Data":"34b5db6899997165077ee526f805d9f6e08b150bc0a57ecaa17c16d494f81c18"}
Sep 18 19:24:34.262693 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.262669    2582 generic.go:356] "Generic (PLEG): container finished" podID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerID="25260587b78eec9d6760247ef17897fbe760f11ca31b8216def03d0cee22aa2f" exitCode=0
Sep 18 19:24:34.262828 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.262757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm" event={"ID":"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3","Type":"ContainerDied","Data":"25260587b78eec9d6760247ef17897fbe760f11ca31b8216def03d0cee22aa2f"}
Sep 18 19:24:34.415325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.415304    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:34.417859 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.417842    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:34.428944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.428925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfqxl\" (UniqueName: \"kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl\") pod \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") "
Sep 18 19:24:34.429010 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.428963    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util\") pod \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") "
Sep 18 19:24:34.429010 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.428994    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle\") pod \"2c1fd185-999c-46a9-ae2c-d26055044aed\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") "
Sep 18 19:24:34.429086 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.429040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util\") pod \"2c1fd185-999c-46a9-ae2c-d26055044aed\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") "
Sep 18 19:24:34.429086 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.429075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mx2c\" (UniqueName: \"kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c\") pod \"2c1fd185-999c-46a9-ae2c-d26055044aed\" (UID: \"2c1fd185-999c-46a9-ae2c-d26055044aed\") "
Sep 18 19:24:34.429185 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.429100    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle\") pod \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\" (UID: \"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa\") "
Sep 18 19:24:34.430875 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.429655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle" (OuterVolumeSpecName: "bundle") pod "2c1fd185-999c-46a9-ae2c-d26055044aed" (UID: "2c1fd185-999c-46a9-ae2c-d26055044aed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:34.430875 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.430177    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle" (OuterVolumeSpecName: "bundle") pod "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" (UID: "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:34.431284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.431263    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c" (OuterVolumeSpecName: "kube-api-access-5mx2c") pod "2c1fd185-999c-46a9-ae2c-d26055044aed" (UID: "2c1fd185-999c-46a9-ae2c-d26055044aed"). InnerVolumeSpecName "kube-api-access-5mx2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:34.431472 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.431452    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl" (OuterVolumeSpecName: "kube-api-access-hfqxl") pod "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" (UID: "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa"). InnerVolumeSpecName "kube-api-access-hfqxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:34.434969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.434944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util" (OuterVolumeSpecName: "util") pod "2c1fd185-999c-46a9-ae2c-d26055044aed" (UID: "2c1fd185-999c-46a9-ae2c-d26055044aed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:34.435267 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.435247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util" (OuterVolumeSpecName: "util") pod "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" (UID: "e11caead-a5ea-4f4b-a1fa-a34ffeb094fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:34.529771 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529710    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:34.529771 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529771    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mx2c\" (UniqueName: \"kubernetes.io/projected/2c1fd185-999c-46a9-ae2c-d26055044aed-kube-api-access-5mx2c\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:34.529921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529788    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:34.529921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529800    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqxl\" (UniqueName: \"kubernetes.io/projected/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-kube-api-access-hfqxl\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:34.529921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529811    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e11caead-a5ea-4f4b-a1fa-a34ffeb094fa-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:34.529921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:34.529822    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c1fd185-999c-46a9-ae2c-d26055044aed-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.269345 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.269317    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg"
Sep 18 19:24:35.269345 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.269324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8zqg" event={"ID":"2c1fd185-999c-46a9-ae2c-d26055044aed","Type":"ContainerDied","Data":"0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14"}
Sep 18 19:24:35.269812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.269371    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ecf7597019865dcf784e155f5cfae31fd33405708830b379204ce8049e2bb14"
Sep 18 19:24:35.271393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.271374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x"
Sep 18 19:24:35.271480 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.271412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5vl9x" event={"ID":"e11caead-a5ea-4f4b-a1fa-a34ffeb094fa","Type":"ContainerDied","Data":"619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13"}
Sep 18 19:24:35.271480 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.271447    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619445b09c1fce7d91631ca43a5ae29631fadb403f183eefc98e4a869ac32e13"
Sep 18 19:24:35.415434 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.415416    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:35.417495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.417479    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:35.436093 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util\") pod \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") "
Sep 18 19:24:35.436138 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle\") pod \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") "
Sep 18 19:24:35.436182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlscw\" (UniqueName: \"kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw\") pod \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") "
Sep 18 19:24:35.436182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle\") pod \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\" (UID: \"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3\") "
Sep 18 19:24:35.436265 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmwx\" (UniqueName: \"kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx\") pod \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") "
Sep 18 19:24:35.436265 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436211    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util\") pod \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\" (UID: \"1e09b2cc-f21f-40e7-942c-8af99bb60ec3\") "
Sep 18 19:24:35.436648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436612    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle" (OuterVolumeSpecName: "bundle") pod "1e09b2cc-f21f-40e7-942c-8af99bb60ec3" (UID: "1e09b2cc-f21f-40e7-942c-8af99bb60ec3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:35.436648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.436624    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle" (OuterVolumeSpecName: "bundle") pod "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" (UID: "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:35.438483 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.438454    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw" (OuterVolumeSpecName: "kube-api-access-jlscw") pod "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" (UID: "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3"). InnerVolumeSpecName "kube-api-access-jlscw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:35.438483 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.438463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx" (OuterVolumeSpecName: "kube-api-access-jgmwx") pod "1e09b2cc-f21f-40e7-942c-8af99bb60ec3" (UID: "1e09b2cc-f21f-40e7-942c-8af99bb60ec3"). InnerVolumeSpecName "kube-api-access-jgmwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:35.443249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.443177    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util" (OuterVolumeSpecName: "util") pod "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" (UID: "945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:35.443464 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.443443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util" (OuterVolumeSpecName: "util") pod "1e09b2cc-f21f-40e7-942c-8af99bb60ec3" (UID: "1e09b2cc-f21f-40e7-942c-8af99bb60ec3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537669    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537690    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537698    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlscw\" (UniqueName: \"kubernetes.io/projected/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-kube-api-access-jlscw\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537709    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3-bundle\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537717    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmwx\" (UniqueName: \"kubernetes.io/projected/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-kube-api-access-jgmwx\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:35.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:35.537736    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e09b2cc-f21f-40e7-942c-8af99bb60ec3-util\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:24:36.277183 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.277151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg"
Sep 18 19:24:36.277608 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.277153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr7kg" event={"ID":"1e09b2cc-f21f-40e7-942c-8af99bb60ec3","Type":"ContainerDied","Data":"02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399"}
Sep 18 19:24:36.277608 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.277273    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02584e5a5dd601b927ade7cc2b68af0cdedaa53681184780886dc1ab917b1399"
Sep 18 19:24:36.279234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.279208    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm"
Sep 18 19:24:36.279234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.279215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q7znm" event={"ID":"945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3","Type":"ContainerDied","Data":"f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3"}
Sep 18 19:24:36.279422 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:36.279254    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d4fa385cd01dba41a621d54aef1d67a0f778b6752d6fe366225aae63b101c3"
Sep 18 19:24:45.540275 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540193    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"]
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540452    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540462    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540474    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540479    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540486    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540491    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540502    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540507    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540512    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540518    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540522    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540527    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540535    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540545    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540550    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540570    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540576    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540582    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540595    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="pull"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540612    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="util"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="945a9d6c-8a0b-4f78-9855-8fb1bdfbaec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540662    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e09b2cc-f21f-40e7-942c-8af99bb60ec3" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11caead-a5ea-4f4b-a1fa-a34ffeb094fa" containerName="extract"
Sep 18 19:24:45.540722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.540675    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1fd185-999c-46a9-ae2c-d26055044aed" containerName="extract"
Sep 18 19:24:45.544528 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.544509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:45.547453 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.547431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:24:45.547536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.547523    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 19:24:45.548073 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.548058    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:24:45.548160 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.548097    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5sm7v\""
Sep 18 19:24:45.570765 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.570746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"]
Sep 18 19:24:45.614800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.614768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46ml\" (UniqueName: \"kubernetes.io/projected/d6a36572-2dfa-4cbf-92f0-29076a421408-kube-api-access-p46ml\") pod \"dns-operator-controller-manager-65dc8f76f5-kbnw5\" (UID: \"d6a36572-2dfa-4cbf-92f0-29076a421408\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:45.715503 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.715465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46ml\" (UniqueName: \"kubernetes.io/projected/d6a36572-2dfa-4cbf-92f0-29076a421408-kube-api-access-p46ml\") pod \"dns-operator-controller-manager-65dc8f76f5-kbnw5\" (UID: \"d6a36572-2dfa-4cbf-92f0-29076a421408\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:45.733783 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.733758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46ml\" (UniqueName: \"kubernetes.io/projected/d6a36572-2dfa-4cbf-92f0-29076a421408-kube-api-access-p46ml\") pod \"dns-operator-controller-manager-65dc8f76f5-kbnw5\" (UID: \"d6a36572-2dfa-4cbf-92f0-29076a421408\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:45.854026 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:45.853947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:46.136585 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:46.136564    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"]
Sep 18 19:24:46.138541 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:46.138516    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a36572_2dfa_4cbf_92f0_29076a421408.slice/crio-f8ba6f19e2bee4b53c474b57676ad3ffc324f23031c2ce7f5edb108e92838d47 WatchSource:0}: Error finding container f8ba6f19e2bee4b53c474b57676ad3ffc324f23031c2ce7f5edb108e92838d47: Status 404 returned error can't find the container with id f8ba6f19e2bee4b53c474b57676ad3ffc324f23031c2ce7f5edb108e92838d47
Sep 18 19:24:46.321487 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:46.321454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5" event={"ID":"d6a36572-2dfa-4cbf-92f0-29076a421408","Type":"ContainerStarted","Data":"f8ba6f19e2bee4b53c474b57676ad3ffc324f23031c2ce7f5edb108e92838d47"}
Sep 18 19:24:48.333944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:48.333844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5" event={"ID":"d6a36572-2dfa-4cbf-92f0-29076a421408","Type":"ContainerStarted","Data":"82489073bb6633b1a7c9a3a9bedb004efb7ae444a505666f36931f0fb46b2b26"}
Sep 18 19:24:48.334356 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:48.334013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:24:48.362877 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:48.362816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5" podStartSLOduration=1.5178271269999999 podStartE2EDuration="3.362803173s" totalImagesPullingTime="1.844976046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:48.362026512 +0000 UTC m=+620.599153499" watchObservedRunningTime="2026-09-18 19:24:48.362803173 +0000 UTC m=+620.599930160"
Sep 18 19:24:50.485549 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.485517    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"]
Sep 18 19:24:50.489063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.489046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.492869 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.492850    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fmv95\""
Sep 18 19:24:50.514761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.514736    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"]
Sep 18 19:24:50.555641 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.555611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.555641 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.555644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6sz\" (UniqueName: \"kubernetes.io/projected/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-kube-api-access-qw6sz\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.656760 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.656718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.656760 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.656762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6sz\" (UniqueName: \"kubernetes.io/projected/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-kube-api-access-qw6sz\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.657100 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.657080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.669418 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.669391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6sz\" (UniqueName: \"kubernetes.io/projected/9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5-kube-api-access-qw6sz\") pod \"kuadrant-operator-controller-manager-6d4cb79c-5qmvp\" (UID: \"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:50.799344 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:50.799264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:51.083785 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:51.083760    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"]
Sep 18 19:24:51.084907 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:24:51.084883    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4d76ae_3bc5_4874_89f6_5cbeb064a8d5.slice/crio-524361300d1c9ca54702d3f6f36d8e6dbbbf6d5875f93a980f199475ce742bcb WatchSource:0}: Error finding container 524361300d1c9ca54702d3f6f36d8e6dbbbf6d5875f93a980f199475ce742bcb: Status 404 returned error can't find the container with id 524361300d1c9ca54702d3f6f36d8e6dbbbf6d5875f93a980f199475ce742bcb
Sep 18 19:24:51.352273 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:51.352187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp" event={"ID":"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5","Type":"ContainerStarted","Data":"524361300d1c9ca54702d3f6f36d8e6dbbbf6d5875f93a980f199475ce742bcb"}
Sep 18 19:24:55.373397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:55.373350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp" event={"ID":"9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5","Type":"ContainerStarted","Data":"6811f4a4cd063eb722233c9f8d248ccbd6df4eef099d8b8f92d695e20b1dc983"}
Sep 18 19:24:55.373869 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:55.373419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:24:55.398863 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:55.398818    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp" podStartSLOduration=1.7284594260000001 podStartE2EDuration="5.398803872s" totalImagesPullingTime="3.670344446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:55.397180666 +0000 UTC m=+627.634307652" watchObservedRunningTime="2026-09-18 19:24:55.398803872 +0000 UTC m=+627.635930859"
Sep 18 19:24:59.340626 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:24:59.340593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-kbnw5"
Sep 18 19:25:06.380765 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:06.380710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-5qmvp"
Sep 18 19:25:42.851492 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.851457    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:42.853678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.853657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:42.856074 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.856043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xk2h6\""
Sep 18 19:25:42.858805 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.858782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjtkh\" (UniqueName: \"kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh\") pod \"authorino-7bcfd8f9cd-qh6tp\" (UID: \"d886680d-7ed5-4794-8bae-149c6645220a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:42.863854 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.863834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:42.959864 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.959832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjtkh\" (UniqueName: \"kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh\") pod \"authorino-7bcfd8f9cd-qh6tp\" (UID: \"d886680d-7ed5-4794-8bae-149c6645220a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:42.969221 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.969203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjtkh\" (UniqueName: \"kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh\") pod \"authorino-7bcfd8f9cd-qh6tp\" (UID: \"d886680d-7ed5-4794-8bae-149c6645220a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:42.989019 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.988995    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:25:42.990964 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:42.990949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:25:43.002466 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.002437    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:25:43.061008 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.060982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zw7m\" (UniqueName: \"kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m\") pod \"authorino-59c974bd48-df276\" (UID: \"7bc5301a-a95a-468d-8b9d-a755db28a370\") " pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:25:43.161967 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.161905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zw7m\" (UniqueName: \"kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m\") pod \"authorino-59c974bd48-df276\" (UID: \"7bc5301a-a95a-468d-8b9d-a755db28a370\") " pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:25:43.162818 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.162801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:43.172431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.172407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zw7m\" (UniqueName: \"kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m\") pod \"authorino-59c974bd48-df276\" (UID: \"7bc5301a-a95a-468d-8b9d-a755db28a370\") " pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:25:43.300215 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.300184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:25:43.435002 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.434975    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:43.436681 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:25:43.436654    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd886680d_7ed5_4794_8bae_149c6645220a.slice/crio-85575ed30170ca90694db1eefbb3cd278070db5c25f9ae07f76c3c0e6fafa2bc WatchSource:0}: Error finding container 85575ed30170ca90694db1eefbb3cd278070db5c25f9ae07f76c3c0e6fafa2bc: Status 404 returned error can't find the container with id 85575ed30170ca90694db1eefbb3cd278070db5c25f9ae07f76c3c0e6fafa2bc
Sep 18 19:25:43.581278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.581255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:25:43.583398 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:25:43.583375    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bc5301a_a95a_468d_8b9d_a755db28a370.slice/crio-75ea2b230d92236041401b184002e147c99750cff6b2cc6548e81e9a36dceec3 WatchSource:0}: Error finding container 75ea2b230d92236041401b184002e147c99750cff6b2cc6548e81e9a36dceec3: Status 404 returned error can't find the container with id 75ea2b230d92236041401b184002e147c99750cff6b2cc6548e81e9a36dceec3
Sep 18 19:25:43.591937 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.591909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-df276" event={"ID":"7bc5301a-a95a-468d-8b9d-a755db28a370","Type":"ContainerStarted","Data":"75ea2b230d92236041401b184002e147c99750cff6b2cc6548e81e9a36dceec3"}
Sep 18 19:25:43.592951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:43.592933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" event={"ID":"d886680d-7ed5-4794-8bae-149c6645220a","Type":"ContainerStarted","Data":"85575ed30170ca90694db1eefbb3cd278070db5c25f9ae07f76c3c0e6fafa2bc"}
Sep 18 19:25:45.607434 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:45.607352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-df276" event={"ID":"7bc5301a-a95a-468d-8b9d-a755db28a370","Type":"ContainerStarted","Data":"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"}
Sep 18 19:25:45.608973 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:45.608948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" event={"ID":"d886680d-7ed5-4794-8bae-149c6645220a","Type":"ContainerStarted","Data":"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"}
Sep 18 19:25:45.622160 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:45.622121    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-df276" podStartSLOduration=1.956732872 podStartE2EDuration="3.622107627s" totalImagesPullingTime="1.665374755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:45.621902629 +0000 UTC m=+677.859029615" watchObservedRunningTime="2026-09-18 19:25:45.622107627 +0000 UTC m=+677.859234612"
Sep 18 19:25:45.636573 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:45.636536    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" podStartSLOduration=1.820040504 podStartE2EDuration="3.636521507s" totalImagesPullingTime="1.816481003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:45.635920027 +0000 UTC m=+677.873047005" watchObservedRunningTime="2026-09-18 19:25:45.636521507 +0000 UTC m=+677.873648495"
Sep 18 19:25:45.669416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:45.669390    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:47.617912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:47.617872    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" podUID="d886680d-7ed5-4794-8bae-149c6645220a" containerName="authorino" containerID="cri-o://0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522" gracePeriod=30
Sep 18 19:25:47.855699 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:47.855678    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:47.900172 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:47.900116    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjtkh\" (UniqueName: \"kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh\") pod \"d886680d-7ed5-4794-8bae-149c6645220a\" (UID: \"d886680d-7ed5-4794-8bae-149c6645220a\") "
Sep 18 19:25:47.902345 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:47.902324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh" (OuterVolumeSpecName: "kube-api-access-mjtkh") pod "d886680d-7ed5-4794-8bae-149c6645220a" (UID: "d886680d-7ed5-4794-8bae-149c6645220a"). InnerVolumeSpecName "kube-api-access-mjtkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:25:48.001284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.001250    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjtkh\" (UniqueName: \"kubernetes.io/projected/d886680d-7ed5-4794-8bae-149c6645220a-kube-api-access-mjtkh\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:25:48.622934 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.622896    2582 generic.go:356] "Generic (PLEG): container finished" podID="d886680d-7ed5-4794-8bae-149c6645220a" containerID="0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522" exitCode=0
Sep 18 19:25:48.623357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.622939    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp"
Sep 18 19:25:48.623357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.622944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" event={"ID":"d886680d-7ed5-4794-8bae-149c6645220a","Type":"ContainerDied","Data":"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"}
Sep 18 19:25:48.623357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.622981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qh6tp" event={"ID":"d886680d-7ed5-4794-8bae-149c6645220a","Type":"ContainerDied","Data":"85575ed30170ca90694db1eefbb3cd278070db5c25f9ae07f76c3c0e6fafa2bc"}
Sep 18 19:25:48.623357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.623003    2582 scope.go:117] "RemoveContainer" containerID="0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"
Sep 18 19:25:48.631448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.631428    2582 scope.go:117] "RemoveContainer" containerID="0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"
Sep 18 19:25:48.631721 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:25:48.631695    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522\": container with ID starting with 0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522 not found: ID does not exist" containerID="0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"
Sep 18 19:25:48.631836 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.631752    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522"} err="failed to get container status \"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522\": rpc error: code = NotFound desc = could not find container \"0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522\": container with ID starting with 0bd68b1589f9adae6b6545541f1f955330ecc9dabeed019d582c01aa2ea27522 not found: ID does not exist"
Sep 18 19:25:48.652345 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.652312    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:48.659535 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:48.659513    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qh6tp"]
Sep 18 19:25:49.894939 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:25:49.894905    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d886680d-7ed5-4794-8bae-149c6645220a" path="/var/lib/kubelet/pods/d886680d-7ed5-4794-8bae-149c6645220a/volumes"
Sep 18 19:26:05.817249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.817215    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5vb74"]
Sep 18 19:26:05.817605 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.817490    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d886680d-7ed5-4794-8bae-149c6645220a" containerName="authorino"
Sep 18 19:26:05.817605 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.817501    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d886680d-7ed5-4794-8bae-149c6645220a" containerName="authorino"
Sep 18 19:26:05.817605 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.817553    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d886680d-7ed5-4794-8bae-149c6645220a" containerName="authorino"
Sep 18 19:26:05.821088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.821068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:05.824314 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.824294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 19:26:05.830779 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.830758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5vb74"]
Sep 18 19:26:05.927189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.927159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cql\" (UniqueName: \"kubernetes.io/projected/d2d24c61-ff61-41fb-ac59-28b59126824c-kube-api-access-p4cql\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:05.927344 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:05.927255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2d24c61-ff61-41fb-ac59-28b59126824c-tls-cert\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.028544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.028516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cql\" (UniqueName: \"kubernetes.io/projected/d2d24c61-ff61-41fb-ac59-28b59126824c-kube-api-access-p4cql\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.028688 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.028558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2d24c61-ff61-41fb-ac59-28b59126824c-tls-cert\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.031081 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.031061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2d24c61-ff61-41fb-ac59-28b59126824c-tls-cert\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.038271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.038251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cql\" (UniqueName: \"kubernetes.io/projected/d2d24c61-ff61-41fb-ac59-28b59126824c-kube-api-access-p4cql\") pod \"authorino-6d9898bd99-5vb74\" (UID: \"d2d24c61-ff61-41fb-ac59-28b59126824c\") " pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.129507 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.129450    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5vb74"
Sep 18 19:26:06.403322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.403292    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5vb74"]
Sep 18 19:26:06.404941 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:26:06.404916    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d24c61_ff61_41fb_ac59_28b59126824c.slice/crio-c8bfb6048a40730d8b7f2f5a0cfad4bf64be9c0bed781688678eb45f14d4f32d WatchSource:0}: Error finding container c8bfb6048a40730d8b7f2f5a0cfad4bf64be9c0bed781688678eb45f14d4f32d: Status 404 returned error can't find the container with id c8bfb6048a40730d8b7f2f5a0cfad4bf64be9c0bed781688678eb45f14d4f32d
Sep 18 19:26:06.707711 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:06.707627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5vb74" event={"ID":"d2d24c61-ff61-41fb-ac59-28b59126824c","Type":"ContainerStarted","Data":"c8bfb6048a40730d8b7f2f5a0cfad4bf64be9c0bed781688678eb45f14d4f32d"}
Sep 18 19:26:07.718795 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:07.718757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5vb74" event={"ID":"d2d24c61-ff61-41fb-ac59-28b59126824c","Type":"ContainerStarted","Data":"14fc92403484d06aa8d60e20fa587d48d14aad2da2088b7aa42cabd77f9ddf43"}
Sep 18 19:26:07.744227 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:07.744177    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-5vb74" podStartSLOduration=1.997570536 podStartE2EDuration="2.744164472s" totalImagesPullingTime="746.593936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:07.742950506 +0000 UTC m=+699.980077492" watchObservedRunningTime="2026-09-18 19:26:07.744164472 +0000 UTC m=+699.981291457"
Sep 18 19:26:07.774166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:07.774137    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:26:07.774330 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:07.774307    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-df276" podUID="7bc5301a-a95a-468d-8b9d-a755db28a370" containerName="authorino" containerID="cri-o://adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8" gracePeriod=30
Sep 18 19:26:08.015288 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.015264    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:26:08.044703 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.044678    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zw7m\" (UniqueName: \"kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m\") pod \"7bc5301a-a95a-468d-8b9d-a755db28a370\" (UID: \"7bc5301a-a95a-468d-8b9d-a755db28a370\") "
Sep 18 19:26:08.046905 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.046874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m" (OuterVolumeSpecName: "kube-api-access-5zw7m") pod "7bc5301a-a95a-468d-8b9d-a755db28a370" (UID: "7bc5301a-a95a-468d-8b9d-a755db28a370"). InnerVolumeSpecName "kube-api-access-5zw7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:26:08.145793 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.145760    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zw7m\" (UniqueName: \"kubernetes.io/projected/7bc5301a-a95a-468d-8b9d-a755db28a370-kube-api-access-5zw7m\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:26:08.725197 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.725161    2582 generic.go:356] "Generic (PLEG): container finished" podID="7bc5301a-a95a-468d-8b9d-a755db28a370" containerID="adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8" exitCode=0
Sep 18 19:26:08.725664 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.725204    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-df276"
Sep 18 19:26:08.725664 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.725244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-df276" event={"ID":"7bc5301a-a95a-468d-8b9d-a755db28a370","Type":"ContainerDied","Data":"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"}
Sep 18 19:26:08.725664 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.725281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-df276" event={"ID":"7bc5301a-a95a-468d-8b9d-a755db28a370","Type":"ContainerDied","Data":"75ea2b230d92236041401b184002e147c99750cff6b2cc6548e81e9a36dceec3"}
Sep 18 19:26:08.725664 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.725297    2582 scope.go:117] "RemoveContainer" containerID="adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"
Sep 18 19:26:08.733979 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.733962    2582 scope.go:117] "RemoveContainer" containerID="adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"
Sep 18 19:26:08.734252 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:26:08.734235    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8\": container with ID starting with adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8 not found: ID does not exist" containerID="adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"
Sep 18 19:26:08.734295 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.734260    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8"} err="failed to get container status \"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8\": rpc error: code = NotFound desc = could not find container \"adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8\": container with ID starting with adc9267aa37fde931ae8cf97ce553660b9df9ac5a54de35f1d35b07d7722b0c8 not found: ID does not exist"
Sep 18 19:26:08.750873 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.750852    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:26:08.757519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:08.757500    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-df276"]
Sep 18 19:26:09.894415 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:26:09.894382    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc5301a-a95a-468d-8b9d-a755db28a370" path="/var/lib/kubelet/pods/7bc5301a-a95a-468d-8b9d-a755db28a370/volumes"
Sep 18 19:28:20.981079 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.981036    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pphkx"]
Sep 18 19:28:20.981485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.981367    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc5301a-a95a-468d-8b9d-a755db28a370" containerName="authorino"
Sep 18 19:28:20.981485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.981382    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc5301a-a95a-468d-8b9d-a755db28a370" containerName="authorino"
Sep 18 19:28:20.981485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.981438    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc5301a-a95a-468d-8b9d-a755db28a370" containerName="authorino"
Sep 18 19:28:20.984363 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.984345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pphkx"
Sep 18 19:28:20.987005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.986982    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:28:20.987005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.986999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:28:20.987820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.987806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:28:20.987899 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.987830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f2gwx\""
Sep 18 19:28:20.987957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.987830    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:28:20.993713 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:20.993694    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pphkx"]
Sep 18 19:28:21.067193 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.067167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g722\" (UniqueName: \"kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.067327 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.067216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.167733 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.167709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g722\" (UniqueName: \"kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.167856 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.167758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.168070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.168055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.177030 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.177008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g722\" (UniqueName: \"kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722\") pod \"s3-init-pphkx\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") " pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.294338 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.294285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pphkx"
Sep 18 19:28:21.570566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.570541    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pphkx"]
Sep 18 19:28:21.573637 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:28:21.573612    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4874c7ab_0fbc_4a61_9c4d_cf8e5f777c80.slice/crio-65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec WatchSource:0}: Error finding container 65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec: Status 404 returned error can't find the container with id 65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec
Sep 18 19:28:21.575626 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:21.575606    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:28:22.329609 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:22.329547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerStarted","Data":"65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec"}
Sep 18 19:28:26.353335 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:26.353294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerStarted","Data":"de636beb30652a1d9d425c80a7596b65a817ad6b5d1aa5bf24c8d2d26503a721"}
Sep 18 19:28:28.363902 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:28.363869    2582 generic.go:356] "Generic (PLEG): container finished" podID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerID="de636beb30652a1d9d425c80a7596b65a817ad6b5d1aa5bf24c8d2d26503a721" exitCode=0
Sep 18 19:28:28.364262 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:28.363943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerDied","Data":"de636beb30652a1d9d425c80a7596b65a817ad6b5d1aa5bf24c8d2d26503a721"}
Sep 18 19:28:32.745363 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:32.745339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:28:33.393799 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:33.393763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerStarted","Data":"b57436a61a89bffc880be8798a516a5dc9ace3c3cea48895622b3ab438dc30d1"}
Sep 18 19:28:33.417854 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:28:33.417809    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pphkx" podStartSLOduration=3.274387972 podStartE2EDuration="13.417795642s" totalImagesPullingTime="8.448349751s" totalInitContainerRuntime="1.695057919s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:28:33.416576885 +0000 UTC m=+845.653703871" watchObservedRunningTime="2026-09-18 19:28:33.417795642 +0000 UTC m=+845.654922628"
Sep 18 19:29:29.796256 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:29.796181    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:29:29.796842 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:29.796816    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:29:29.809291 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:29.809270    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:29:29.809754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:29.809718    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:29:37.691476 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:37.691442    2582 generic.go:356] "Generic (PLEG): container finished" podID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerID="b57436a61a89bffc880be8798a516a5dc9ace3c3cea48895622b3ab438dc30d1" exitCode=0
Sep 18 19:29:37.691871 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:37.691503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerDied","Data":"b57436a61a89bffc880be8798a516a5dc9ace3c3cea48895622b3ab438dc30d1"}
Sep 18 19:29:38.816513 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.816484    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pphkx"
Sep 18 19:29:38.827227 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.827207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache\") pod \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") "
Sep 18 19:29:38.827296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.827266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g722\" (UniqueName: \"kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722\") pod \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\" (UID: \"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80\") "
Sep 18 19:29:38.829456 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.829425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722" (OuterVolumeSpecName: "kube-api-access-7g722") pod "4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" (UID: "4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80"). InnerVolumeSpecName "kube-api-access-7g722". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:29:38.848029 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.848004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache" (OuterVolumeSpecName: "model-cache") pod "4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" (UID: "4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:29:38.928079 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.928051    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:29:38.928157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:38.928081    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g722\" (UniqueName: \"kubernetes.io/projected/4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80-kube-api-access-7g722\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:29:39.702490 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:39.702459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pphkx" event={"ID":"4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80","Type":"ContainerDied","Data":"65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec"}
Sep 18 19:29:39.702490 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:39.702486    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pphkx"
Sep 18 19:29:39.702490 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:29:39.702496    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a16dc681f56b953fdc2b13af423e409a176e0c45bd5f02e6ee63d2142d36ec"
Sep 18 19:30:03.844336 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844301    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:30:03.844820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844592    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerName="download-hf-model"
Sep 18 19:30:03.844820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844603    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerName="download-hf-model"
Sep 18 19:30:03.844820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844611    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerName="s3-init"
Sep 18 19:30:03.844820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844616    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerName="s3-init"
Sep 18 19:30:03.844820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.844664    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80" containerName="s3-init"
Sep 18 19:30:03.846494 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.846478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.850136 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.850106    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Sep 18 19:30:03.850334 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.850313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Sep 18 19:30:03.850401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.850313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Sep 18 19:30:03.850568 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.850549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Sep 18 19:30:03.850916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.850902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-f9zwv\""
Sep 18 19:30:03.866112 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.866081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:30:03.927188 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927258    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4j4s\" (UniqueName: \"kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927373 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:03.927550 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:03.927409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028192 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028192 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4j4s\" (UniqueName: \"kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028705 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028705 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028705 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.028852 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.028833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.030695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.030675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.030986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.030968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.039129 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.039106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4j4s\" (UniqueName: \"kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s\") pod \"cpu-accel-preset-kserve-9788fc657-fzg4s\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.156314 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.156223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:30:04.434577 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.434551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:30:04.437293 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:30:04.437267    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode327a5d1_7321_48e1_a5da_70c9b5958694.slice/crio-06041c796173ec1607546d50067e4f50b8a810a945f30493f55a2d0081882925 WatchSource:0}: Error finding container 06041c796173ec1607546d50067e4f50b8a810a945f30493f55a2d0081882925: Status 404 returned error can't find the container with id 06041c796173ec1607546d50067e4f50b8a810a945f30493f55a2d0081882925
Sep 18 19:30:04.825322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.825277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerStarted","Data":"16242213f8f10219324a03ffdb98f5efff40b73383a21efefd1bcee197c901bd"}
Sep 18 19:30:04.825322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:04.825326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerStarted","Data":"06041c796173ec1607546d50067e4f50b8a810a945f30493f55a2d0081882925"}
Sep 18 19:30:08.846157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:08.846064    2582 generic.go:356] "Generic (PLEG): container finished" podID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerID="16242213f8f10219324a03ffdb98f5efff40b73383a21efefd1bcee197c901bd" exitCode=0
Sep 18 19:30:08.846157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:08.846142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerDied","Data":"16242213f8f10219324a03ffdb98f5efff40b73383a21efefd1bcee197c901bd"}
Sep 18 19:30:34.129855 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:34.129817    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:30:56.126297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:56.126250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerStarted","Data":"3abc8c5b46ddc9fcfbfffc13d58a3c9af122e09f80ff1e25d47a2e5b3aaedc7d"}
Sep 18 19:30:56.153349 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:56.153295    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" podStartSLOduration=2.341202441 podStartE2EDuration="53.153279555s" totalImagesPullingTime="46.756649627s" totalInitContainerRuntime="4.055427487s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:30:56.150991027 +0000 UTC m=+988.388118013" watchObservedRunningTime="2026-09-18 19:30:56.153279555 +0000 UTC m=+988.390406546"
Sep 18 19:30:56.156882 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:30:56.156853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:31:11.153536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:11.153491    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="main" containerID="cri-o://3abc8c5b46ddc9fcfbfffc13d58a3c9af122e09f80ff1e25d47a2e5b3aaedc7d" gracePeriod=45
Sep 18 19:31:56.452952 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.452877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-9788fc657-fzg4s_e327a5d1-7321-48e1-a5da-70c9b5958694/main/0.log"
Sep 18 19:31:56.453338 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.453244    2582 generic.go:356] "Generic (PLEG): container finished" podID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerID="3abc8c5b46ddc9fcfbfffc13d58a3c9af122e09f80ff1e25d47a2e5b3aaedc7d" exitCode=137
Sep 18 19:31:56.453338 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.453318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerDied","Data":"3abc8c5b46ddc9fcfbfffc13d58a3c9af122e09f80ff1e25d47a2e5b3aaedc7d"}
Sep 18 19:31:56.790030 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.790009    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-9788fc657-fzg4s_e327a5d1-7321-48e1-a5da-70c9b5958694/main/0.log"
Sep 18 19:31:56.791706 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.791689    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:31:56.862090 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862189    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4j4s\" (UniqueName: \"kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s\") pod \"e327a5d1-7321-48e1-a5da-70c9b5958694\" (UID: \"e327a5d1-7321-48e1-a5da-70c9b5958694\") "
Sep 18 19:31:56.862532 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862414    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache" (OuterVolumeSpecName: "model-cache") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:56.862668 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.862640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home" (OuterVolumeSpecName: "home") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:56.863315 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.863279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:56.864631 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.864603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s" (OuterVolumeSpecName: "kube-api-access-n4j4s") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "kube-api-access-n4j4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:56.864631 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.864621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm" (OuterVolumeSpecName: "dshm") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:56.864824 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.864805    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:56.918279 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.918247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e327a5d1-7321-48e1-a5da-70c9b5958694" (UID: "e327a5d1-7321-48e1-a5da-70c9b5958694"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:56.963371 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963349    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963374    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963386    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e327a5d1-7321-48e1-a5da-70c9b5958694-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963393    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963400    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963409    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4j4s\" (UniqueName: \"kubernetes.io/projected/e327a5d1-7321-48e1-a5da-70c9b5958694-kube-api-access-n4j4s\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:56.963460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:56.963417    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e327a5d1-7321-48e1-a5da-70c9b5958694-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:31:57.460889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.460861    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-9788fc657-fzg4s_e327a5d1-7321-48e1-a5da-70c9b5958694/main/0.log"
Sep 18 19:31:57.461357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.461314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s" event={"ID":"e327a5d1-7321-48e1-a5da-70c9b5958694","Type":"ContainerDied","Data":"06041c796173ec1607546d50067e4f50b8a810a945f30493f55a2d0081882925"}
Sep 18 19:31:57.461428 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.461365    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"
Sep 18 19:31:57.461486 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.461356    2582 scope.go:117] "RemoveContainer" containerID="3abc8c5b46ddc9fcfbfffc13d58a3c9af122e09f80ff1e25d47a2e5b3aaedc7d"
Sep 18 19:31:57.470312 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.470291    2582 scope.go:117] "RemoveContainer" containerID="16242213f8f10219324a03ffdb98f5efff40b73383a21efefd1bcee197c901bd"
Sep 18 19:31:57.492158 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.492133    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:31:57.499028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.499008    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-9788fc657-fzg4s"]
Sep 18 19:31:57.895244 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:31:57.895209    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" path="/var/lib/kubelet/pods/e327a5d1-7321-48e1-a5da-70c9b5958694/volumes"
Sep 18 19:32:24.318060 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.317988    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:32:24.318462 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.318273    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="main"
Sep 18 19:32:24.318462 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.318283    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="main"
Sep 18 19:32:24.318462 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.318297    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="storage-initializer"
Sep 18 19:32:24.318462 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.318302    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="storage-initializer"
Sep 18 19:32:24.318462 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.318352    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e327a5d1-7321-48e1-a5da-70c9b5958694" containerName="main"
Sep 18 19:32:24.320352 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.320319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.322878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.322857    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 19:32:24.323783 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.323743    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 19:32:24.323783 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.323722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tzmj8\""
Sep 18 19:32:24.323952 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.323757    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 19:32:24.331058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.331039    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:32:24.469263 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.469414 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.469414 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.469414 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.469414 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxdv\" (UniqueName: \"kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.469569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.469435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570166 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxdv\" (UniqueName: \"kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570573 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570626 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.570661 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.570636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.572618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.572598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.572883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.572867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.607965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.607941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxdv\" (UniqueName: \"kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.631936 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.631915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:24.918073 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:24.918048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:32:24.919841 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:32:24.919811    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa3688c_2796_4423_9e14_d0264444ddd1.slice/crio-02d53a7a75a48b711c2f80fcbab6e619fb17bd66751e8048e04a4b6c9aa8408d WatchSource:0}: Error finding container 02d53a7a75a48b711c2f80fcbab6e619fb17bd66751e8048e04a4b6c9aa8408d: Status 404 returned error can't find the container with id 02d53a7a75a48b711c2f80fcbab6e619fb17bd66751e8048e04a4b6c9aa8408d
Sep 18 19:32:25.595357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:25.595313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" event={"ID":"6fa3688c-2796-4423-9e14-d0264444ddd1","Type":"ContainerStarted","Data":"02d53a7a75a48b711c2f80fcbab6e619fb17bd66751e8048e04a4b6c9aa8408d"}
Sep 18 19:32:26.601088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:26.601052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" event={"ID":"6fa3688c-2796-4423-9e14-d0264444ddd1","Type":"ContainerStarted","Data":"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"}
Sep 18 19:32:26.633024 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:26.632985    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:26.634432 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:26.634390    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podStartSLOduration=1.159657652 podStartE2EDuration="2.634374855s" totalImagesPullingTime="1.474717203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:32:26.629313554 +0000 UTC m=+1078.866440540" watchObservedRunningTime="2026-09-18 19:32:26.634374855 +0000 UTC m=+1078.871501840"
Sep 18 19:32:34.632141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:34.632101    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:34.645484 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:34.645455    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:35.644767 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:35.644720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:32:38.707084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.707050    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:32:38.710149 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.710129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.712769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.712720    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Sep 18 19:32:38.712889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.712845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-6t8hx\""
Sep 18 19:32:38.713119 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.713103    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Sep 18 19:32:38.713217 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.713160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Sep 18 19:32:38.713441 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.713426    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Sep 18 19:32:38.731377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.731352    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:32:38.789601 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpkq\" (UniqueName: \"kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.789976 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.789797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpkq\" (UniqueName: \"kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.890921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.890795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.891198 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.891176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.891250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.891219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.891296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.891250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.891296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.891282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.893048 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.893018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.893361 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.893343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:38.900702 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:38.900675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpkq\" (UniqueName: \"kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq\") pod \"cpu-accel-precedence-kserve-697c6d8c-5gjvl\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:39.018932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:39.018907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:32:39.300225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:39.300197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:32:39.302248 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:32:39.302221    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6bbc97_9c31_48b2_8f27_c2c2eec54ad0.slice/crio-18f6156b2c5ce45aca8fab33d48d23168580985b5dbc0d572bab4fdce2c692a8 WatchSource:0}: Error finding container 18f6156b2c5ce45aca8fab33d48d23168580985b5dbc0d572bab4fdce2c692a8: Status 404 returned error can't find the container with id 18f6156b2c5ce45aca8fab33d48d23168580985b5dbc0d572bab4fdce2c692a8
Sep 18 19:32:39.662141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:39.662047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerStarted","Data":"877b3e2d666b6c86b183143ccaf35b1044fe3323eab7b2d034d0f95111ea1f39"}
Sep 18 19:32:39.662141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:39.662105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerStarted","Data":"18f6156b2c5ce45aca8fab33d48d23168580985b5dbc0d572bab4fdce2c692a8"}
Sep 18 19:32:43.686229 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:43.686192    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerID="877b3e2d666b6c86b183143ccaf35b1044fe3323eab7b2d034d0f95111ea1f39" exitCode=0
Sep 18 19:32:43.686688 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:32:43.686264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerDied","Data":"877b3e2d666b6c86b183143ccaf35b1044fe3323eab7b2d034d0f95111ea1f39"}
Sep 18 19:33:01.568088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:01.568049    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:33:03.002132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:03.002089    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:33:03.012613 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:33:03.012575    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:33:03.012613 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:33:03.012613 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:33:03.012613 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e" cmd=["/bin/sleep","15"]
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:03.012636    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" message="" err=<
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:33:03.012648    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" containerID="cri-o://332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"
Sep 18 19:33:03.012951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:03.012697    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" containerID="cri-o://332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e" gracePeriod=60
Sep 18 19:33:03.646157 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:03.646047    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:04.645757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:04.645702    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:04.646287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:04.645842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:33:05.645611 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:05.645554    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:06.645578 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:06.645495    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:07.645947 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:07.645904    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:08.645662 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:08.645609    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:09.646158 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:09.646109    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:10.645585 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:10.645538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 18 19:33:11.685797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.685714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:33:11.788450 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788340    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.788633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.788633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbxdv\" (UniqueName: \"kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.788633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788572    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.788633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home" (OuterVolumeSpecName: "home") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:11.788633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.789172 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir\") pod \"6fa3688c-2796-4423-9e14-d0264444ddd1\" (UID: \"6fa3688c-2796-4423-9e14-d0264444ddd1\") "
Sep 18 19:33:11.789172 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788908    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.789172 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.788978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:11.789172 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.789085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:11.791572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.791548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv" (OuterVolumeSpecName: "kube-api-access-lbxdv") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "kube-api-access-lbxdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:33:11.791889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.791866    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm" (OuterVolumeSpecName: "dshm") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:33:11.791990 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.791920    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fa3688c-2796-4423-9e14-d0264444ddd1" (UID: "6fa3688c-2796-4423-9e14-d0264444ddd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:33:11.860122 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.860084    2582 generic.go:356] "Generic (PLEG): container finished" podID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerID="332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e" exitCode=0
Sep 18 19:33:11.860122 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.860130    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"
Sep 18 19:33:11.860387 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.860145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" event={"ID":"6fa3688c-2796-4423-9e14-d0264444ddd1","Type":"ContainerDied","Data":"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"}
Sep 18 19:33:11.860387 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.860201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" event={"ID":"6fa3688c-2796-4423-9e14-d0264444ddd1","Type":"ContainerDied","Data":"02d53a7a75a48b711c2f80fcbab6e619fb17bd66751e8048e04a4b6c9aa8408d"}
Sep 18 19:33:11.860387 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.860231    2582 scope.go:117] "RemoveContainer" containerID="332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"
Sep 18 19:33:11.889656 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.889624    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.889656 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.889657    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.889904 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.889672    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxdv\" (UniqueName: \"kubernetes.io/projected/6fa3688c-2796-4423-9e14-d0264444ddd1-kube-api-access-lbxdv\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.889904 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.889689    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3688c-2796-4423-9e14-d0264444ddd1-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.889904 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.889703    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3688c-2796-4423-9e14-d0264444ddd1-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:33:11.897774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.897746    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:33:11.897897 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:11.897786    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5"]
Sep 18 19:33:12.645613 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:12.645567    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-c5hj5" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:33:12.936465 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:12.936286    2582 scope.go:117] "RemoveContainer" containerID="332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"
Sep 18 19:33:12.936750 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:33:12.936648    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e\": container with ID starting with 332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e not found: ID does not exist" containerID="332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"
Sep 18 19:33:12.936750 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:12.936679    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e"} err="failed to get container status \"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e\": rpc error: code = NotFound desc = could not find container \"332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e\": container with ID starting with 332811049d9ee0ff35b5cc416929d5ccade5605b0214b2cd5d2f1706210f027e not found: ID does not exist"
Sep 18 19:33:13.872139 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:13.872096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerStarted","Data":"5a680bf42494b496ef98f1b719c6ad42db3b32b5a7f85ca5d1a06e8f7defd5a5"}
Sep 18 19:33:13.895139 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:13.895109    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" path="/var/lib/kubelet/pods/6fa3688c-2796-4423-9e14-d0264444ddd1/volumes"
Sep 18 19:33:13.909923 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:13.909868    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" podStartSLOduration=2.22597188 podStartE2EDuration="35.909852623s" totalImagesPullingTime="29.581506608s" totalInitContainerRuntime="4.102374135s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:33:13.908182025 +0000 UTC m=+1126.145309012" watchObservedRunningTime="2026-09-18 19:33:13.909852623 +0000 UTC m=+1126.146979609"
Sep 18 19:33:14.019626 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:14.019572    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:33:28.894685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:33:28.894637    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="main" containerID="cri-o://5a680bf42494b496ef98f1b719c6ad42db3b32b5a7f85ca5d1a06e8f7defd5a5" gracePeriod=45
Sep 18 19:34:01.072929 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.072856    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:34:01.073354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.073178    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main"
Sep 18 19:34:01.073354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.073189    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main"
Sep 18 19:34:01.073354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.073256    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa3688c-2796-4423-9e14-d0264444ddd1" containerName="main"
Sep 18 19:34:01.075311 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.075291    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.077956 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.077936    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 19:34:01.078053 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.078001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9jv9p\""
Sep 18 19:34:01.078865 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.078844    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 19:34:01.078970 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.078918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 19:34:01.078970 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.078932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-6xxsg\""
Sep 18 19:34:01.093405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.093384    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:34:01.186881 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.186843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjhl\" (UniqueName: \"kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.187061 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.186983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.287499 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.287467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.287683 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.287526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjhl\" (UniqueName: \"kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.290395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.290368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.297854 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.297830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjhl\" (UniqueName: \"kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.386106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.386014    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:01.664398 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.664322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:34:01.668603 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:34:01.668561    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb392bd0f_b9ca_464e_9801_5881413347d6.slice/crio-67ba3b7ccc2740df832b236585d673c33530d799be08a1a47d8ed3962e378936 WatchSource:0}: Error finding container 67ba3b7ccc2740df832b236585d673c33530d799be08a1a47d8ed3962e378936: Status 404 returned error can't find the container with id 67ba3b7ccc2740df832b236585d673c33530d799be08a1a47d8ed3962e378936
Sep 18 19:34:01.670968 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:01.670948    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:34:02.112174 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:02.112130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" event={"ID":"b392bd0f-b9ca-464e-9801-5881413347d6","Type":"ContainerStarted","Data":"67ba3b7ccc2740df832b236585d673c33530d799be08a1a47d8ed3962e378936"}
Sep 18 19:34:05.130309 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:05.130279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" event={"ID":"b392bd0f-b9ca-464e-9801-5881413347d6","Type":"ContainerStarted","Data":"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"}
Sep 18 19:34:05.130716 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:05.130384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:34:05.165753 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:05.161145    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podStartSLOduration=1.7476420780000002 podStartE2EDuration="4.161126738s" totalImagesPullingTime="2.41348466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:05.158247681 +0000 UTC m=+1177.395374670" watchObservedRunningTime="2026-09-18 19:34:05.161126738 +0000 UTC m=+1177.398253725"
Sep 18 19:34:14.173141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.173073    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-697c6d8c-5gjvl_fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0/main/0.log"
Sep 18 19:34:14.173519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.173436    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerID="5a680bf42494b496ef98f1b719c6ad42db3b32b5a7f85ca5d1a06e8f7defd5a5" exitCode=137
Sep 18 19:34:14.173519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.173473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerDied","Data":"5a680bf42494b496ef98f1b719c6ad42db3b32b5a7f85ca5d1a06e8f7defd5a5"}
Sep 18 19:34:14.536869 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.536847    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-697c6d8c-5gjvl_fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0/main/0.log"
Sep 18 19:34:14.539007 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.538988    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:34:14.588467 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588496    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlpkq\" (UniqueName: \"kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588688 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588566    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588688 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588646    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs\") pod \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\" (UID: \"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0\") "
Sep 18 19:34:14.588809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:14.588983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.588960    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.589057 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.589028    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home" (OuterVolumeSpecName: "home") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:14.589130 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.589078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:14.590858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.590837    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:34:14.590963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.590912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq" (OuterVolumeSpecName: "kube-api-access-jlpkq") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "kube-api-access-jlpkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:34:14.591017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.590962    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm" (OuterVolumeSpecName: "dshm") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:14.648674 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.648614    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" (UID: "fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:14.689459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689432    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.689459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689455    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.689595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689464    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpkq\" (UniqueName: \"kubernetes.io/projected/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kube-api-access-jlpkq\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.689595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689492    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.689595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689500    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:14.689595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:14.689507    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:34:15.179411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.179385    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-697c6d8c-5gjvl_fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0/main/0.log"
Sep 18 19:34:15.179840 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.179784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl" event={"ID":"fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0","Type":"ContainerDied","Data":"18f6156b2c5ce45aca8fab33d48d23168580985b5dbc0d572bab4fdce2c692a8"}
Sep 18 19:34:15.179840 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.179827    2582 scope.go:117] "RemoveContainer" containerID="5a680bf42494b496ef98f1b719c6ad42db3b32b5a7f85ca5d1a06e8f7defd5a5"
Sep 18 19:34:15.179922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.179858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"
Sep 18 19:34:15.189155 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.189138    2582 scope.go:117] "RemoveContainer" containerID="877b3e2d666b6c86b183143ccaf35b1044fe3323eab7b2d034d0f95111ea1f39"
Sep 18 19:34:15.202256 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.202235    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:34:15.205629 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.205610    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-697c6d8c-5gjvl"]
Sep 18 19:34:15.895873 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:15.895838    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" path="/var/lib/kubelet/pods/fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0/volumes"
Sep 18 19:34:29.833457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:29.833427    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:34:29.835877 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:29.835854    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:34:29.840096 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:29.840076    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:34:29.842938 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:29.842923    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:34:35.952772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.952721    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:34:35.953231 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.953209    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="main"
Sep 18 19:34:35.953278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.953237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="main"
Sep 18 19:34:35.953278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.953259    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="storage-initializer"
Sep 18 19:34:35.953278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.953270    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="storage-initializer"
Sep 18 19:34:35.953371 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.953358    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6bbc97-9c31-48b2-8f27-c2c2eec54ad0" containerName="main"
Sep 18 19:34:35.959336 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.959316    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:35.962008 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.961983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Sep 18 19:34:35.962225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.962203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:34:35.962527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.962507    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 18 19:34:35.962646 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.962629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 18 19:34:35.962917 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.962897    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 18 19:34:35.963257 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.963238    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-g2ltm\""
Sep 18 19:34:35.978448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:35.978424    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:34:36.049154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvrb\" (UniqueName: \"kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049255 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.049460 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.049378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.137534 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.137507    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:34:36.149620 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149795 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149795 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvrb\" (UniqueName: \"kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.149884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.149858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.150101 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.150080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.150354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.150333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.150431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.150368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.150431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.150398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.150540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.150494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.152191 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.152172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.152337 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.152319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.172203 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.172178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvrb\" (UniqueName: \"kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.268971 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.268941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:36.584158 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:36.584071    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:34:36.589749 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:34:36.589705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666a78e2_53da_47bf_accf_7479f67fd260.slice/crio-f9ed29a2d6d4b5db1fe79ae631d6cb696896c9cb187066b5b707ff0f5f821680 WatchSource:0}: Error finding container f9ed29a2d6d4b5db1fe79ae631d6cb696896c9cb187066b5b707ff0f5f821680: Status 404 returned error can't find the container with id f9ed29a2d6d4b5db1fe79ae631d6cb696896c9cb187066b5b707ff0f5f821680
Sep 18 19:34:37.293003 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:37.292961    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerStarted","Data":"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"}
Sep 18 19:34:37.293470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:37.293018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerStarted","Data":"f9ed29a2d6d4b5db1fe79ae631d6cb696896c9cb187066b5b707ff0f5f821680"}
Sep 18 19:34:39.304372 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:39.304337    2582 generic.go:356] "Generic (PLEG): container finished" podID="666a78e2-53da-47bf-accf-7479f67fd260" containerID="9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c" exitCode=0
Sep 18 19:34:39.304759 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:39.304378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerDied","Data":"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"}
Sep 18 19:34:40.310941 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:40.310906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerStarted","Data":"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"}
Sep 18 19:34:40.345403 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:40.345347    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podStartSLOduration=3.205373851 podStartE2EDuration="5.345327465s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.139953614s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:40.343497344 +0000 UTC m=+1212.580624330" watchObservedRunningTime="2026-09-18 19:34:40.345327465 +0000 UTC m=+1212.582454452"
Sep 18 19:34:41.269742 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:41.269685    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:46.137648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:46.137614    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:34:46.269885 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:46.269849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:34:46.271537 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:46.271509    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:34:56.137949 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:56.137912    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:34:56.269696 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:34:56.269655    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:06.137779 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:06.137741    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:06.270396 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:06.270357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:12.439927 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:12.439838    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:35:12.441695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:12.441663    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:16.270034 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:16.269987    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:22.441912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:22.441878    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:26.269565 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:26.269525    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:27.459239 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:27.459193    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" containerID="cri-o://bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f" gracePeriod=45
Sep 18 19:35:32.441742 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:32.441690    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:36.269826 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:36.269789    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:42.442317 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:42.442279    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:46.270149 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:46.270102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:35:52.442492 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:52.442445    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:35:56.278894 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:56.278858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:35:56.285971 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:56.285948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:35:57.701784 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.701756    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:35:57.703342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.703316    2582 generic.go:356] "Generic (PLEG): container finished" podID="b392bd0f-b9ca-464e-9801-5881413347d6" containerID="bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f" exitCode=0
Sep 18 19:35:57.703496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.703396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" event={"ID":"b392bd0f-b9ca-464e-9801-5881413347d6","Type":"ContainerDied","Data":"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"}
Sep 18 19:35:57.703496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.703448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr" event={"ID":"b392bd0f-b9ca-464e-9801-5881413347d6","Type":"ContainerDied","Data":"67ba3b7ccc2740df832b236585d673c33530d799be08a1a47d8ed3962e378936"}
Sep 18 19:35:57.703496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.703469    2582 scope.go:117] "RemoveContainer" containerID="bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"
Sep 18 19:35:57.711809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.711758    2582 scope.go:117] "RemoveContainer" containerID="bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"
Sep 18 19:35:57.712063 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:35:57.712040    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f\": container with ID starting with bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f not found: ID does not exist" containerID="bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"
Sep 18 19:35:57.712156 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.712070    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f"} err="failed to get container status \"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f\": rpc error: code = NotFound desc = could not find container \"bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f\": container with ID starting with bab87c0358235530ca06a3513d0f1205053d437648ba3ccf6595ce53d1fb170f not found: ID does not exist"
Sep 18 19:35:57.798995 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.798386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs\") pod \"b392bd0f-b9ca-464e-9801-5881413347d6\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") "
Sep 18 19:35:57.798995 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.798445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhjhl\" (UniqueName: \"kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl\") pod \"b392bd0f-b9ca-464e-9801-5881413347d6\" (UID: \"b392bd0f-b9ca-464e-9801-5881413347d6\") "
Sep 18 19:35:57.801556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.801525    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b392bd0f-b9ca-464e-9801-5881413347d6" (UID: "b392bd0f-b9ca-464e-9801-5881413347d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:35:57.801702 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.801569    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl" (OuterVolumeSpecName: "kube-api-access-bhjhl") pod "b392bd0f-b9ca-464e-9801-5881413347d6" (UID: "b392bd0f-b9ca-464e-9801-5881413347d6"). InnerVolumeSpecName "kube-api-access-bhjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:35:57.903399 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.903358    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b392bd0f-b9ca-464e-9801-5881413347d6-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:35:57.903399 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:57.903398    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjhl\" (UniqueName: \"kubernetes.io/projected/b392bd0f-b9ca-464e-9801-5881413347d6-kube-api-access-bhjhl\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:35:58.709432 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:58.709395    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"
Sep 18 19:35:58.726511 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:58.726476    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:35:58.732882 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:58.732854    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wvwqr"]
Sep 18 19:35:59.895711 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:35:59.895669    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" path="/var/lib/kubelet/pods/b392bd0f-b9ca-464e-9801-5881413347d6/volumes"
Sep 18 19:36:00.691377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:00.691343    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:36:09.293804 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.293772    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:09.294156 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.294062    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main"
Sep 18 19:36:09.294156 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.294073    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main"
Sep 18 19:36:09.294156 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.294128    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b392bd0f-b9ca-464e-9801-5881413347d6" containerName="main"
Sep 18 19:36:09.298747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.298714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.302038 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.302014    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 19:36:09.302038 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.302025    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 19:36:09.302038 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.302041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 19:36:09.302281 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.302026    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-hhn8n\""
Sep 18 19:36:09.307482 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.307464    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:09.397442 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.397658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.397658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.397658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.397658 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgbz\" (UniqueName: \"kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.397883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.397669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498687 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.498916 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.498901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgbz\" (UniqueName: \"kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.499226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.499077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.499226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.499129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.499226 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.499176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.501205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.501182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.501437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.501421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.507320 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.507284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgbz\" (UniqueName: \"kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vrfrr\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.609794 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.609688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:09.885349 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:09.885323    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:09.887592 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:36:09.887565    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a9537d_8161_4f3a_a662_e40a6ad648f9.slice/crio-28d4b79709e4331c7f60664a7bd9db393890c3b5c6d99d66e2a23fcd2b9ad956 WatchSource:0}: Error finding container 28d4b79709e4331c7f60664a7bd9db393890c3b5c6d99d66e2a23fcd2b9ad956: Status 404 returned error can't find the container with id 28d4b79709e4331c7f60664a7bd9db393890c3b5c6d99d66e2a23fcd2b9ad956
Sep 18 19:36:10.772257 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:10.772223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" event={"ID":"39a9537d-8161-4f3a-a662-e40a6ad648f9","Type":"ContainerStarted","Data":"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"}
Sep 18 19:36:10.772645 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:10.772271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" event={"ID":"39a9537d-8161-4f3a-a662-e40a6ad648f9","Type":"ContainerStarted","Data":"28d4b79709e4331c7f60664a7bd9db393890c3b5c6d99d66e2a23fcd2b9ad956"}
Sep 18 19:36:10.794030 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:10.793974    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" podStartSLOduration=1.7939586090000001 podStartE2EDuration="1.793958609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:10.791680729 +0000 UTC m=+1303.028807715" watchObservedRunningTime="2026-09-18 19:36:10.793958609 +0000 UTC m=+1303.031085594"
Sep 18 19:36:11.610788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:11.610754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:15.712135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:15.712073    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" containerID="cri-o://556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6" gracePeriod=45
Sep 18 19:36:16.280145 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.280105    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 18 19:36:16.556186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.556163    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:36:16.651108 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651115    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggvrb\" (UniqueName: \"kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651580 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm\") pod \"666a78e2-53da-47bf-accf-7479f67fd260\" (UID: \"666a78e2-53da-47bf-accf-7479f67fd260\") "
Sep 18 19:36:16.651580 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651559    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:16.651891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.651869    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/666a78e2-53da-47bf-accf-7479f67fd260-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.652162 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.652108    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home" (OuterVolumeSpecName: "home") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.652162 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.652121    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache" (OuterVolumeSpecName: "model-cache") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.653595 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.653561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:16.653761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.653721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm" (OuterVolumeSpecName: "dshm") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.653887 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.653867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb" (OuterVolumeSpecName: "kube-api-access-ggvrb") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "kube-api-access-ggvrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:16.664741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.664703    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.674244 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.674221    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "666a78e2-53da-47bf-accf-7479f67fd260" (UID: "666a78e2-53da-47bf-accf-7479f67fd260"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.752403 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752374    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvrb\" (UniqueName: \"kubernetes.io/projected/666a78e2-53da-47bf-accf-7479f67fd260-kube-api-access-ggvrb\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752403 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752400    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752415    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752427    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752438    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666a78e2-53da-47bf-accf-7479f67fd260-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752448    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.752761 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.752461    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/666a78e2-53da-47bf-accf-7479f67fd260-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.802567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.802516    2582 generic.go:356] "Generic (PLEG): container finished" podID="666a78e2-53da-47bf-accf-7479f67fd260" containerID="556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6" exitCode=0
Sep 18 19:36:16.802654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.802569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"
Sep 18 19:36:16.802654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.802603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerDied","Data":"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"}
Sep 18 19:36:16.802654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.802645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5" event={"ID":"666a78e2-53da-47bf-accf-7479f67fd260","Type":"ContainerDied","Data":"f9ed29a2d6d4b5db1fe79ae631d6cb696896c9cb187066b5b707ff0f5f821680"}
Sep 18 19:36:16.802793 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.802662    2582 scope.go:117] "RemoveContainer" containerID="556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"
Sep 18 19:36:16.811938 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.811922    2582 scope.go:117] "RemoveContainer" containerID="9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"
Sep 18 19:36:16.819431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.819412    2582 scope.go:117] "RemoveContainer" containerID="556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"
Sep 18 19:36:16.819695 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:36:16.819676    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6\": container with ID starting with 556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6 not found: ID does not exist" containerID="556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"
Sep 18 19:36:16.819757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.819702    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6"} err="failed to get container status \"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6\": rpc error: code = NotFound desc = could not find container \"556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6\": container with ID starting with 556895cacb25988dbbd1c2b3a9aba8867a9a1fb14b51cfb9cf3a3e43e156c6d6 not found: ID does not exist"
Sep 18 19:36:16.819757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.819719    2582 scope.go:117] "RemoveContainer" containerID="9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"
Sep 18 19:36:16.820066 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:36:16.820018    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c\": container with ID starting with 9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c not found: ID does not exist" containerID="9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"
Sep 18 19:36:16.820066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.820041    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c"} err="failed to get container status \"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c\": rpc error: code = NotFound desc = could not find container \"9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c\": container with ID starting with 9df6b1d11bebcdd37af1eeede808fd6b2bebf6fdf06923c684a07bc0b405f10c not found: ID does not exist"
Sep 18 19:36:16.824741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.824707    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:36:16.829406 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:16.829388    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6ffc59b94ftmvq5"]
Sep 18 19:36:17.895350 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:17.895322    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="666a78e2-53da-47bf-accf-7479f67fd260" path="/var/lib/kubelet/pods/666a78e2-53da-47bf-accf-7479f67fd260/volumes"
Sep 18 19:36:19.610608 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:19.610571    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:19.622903 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:19.622880    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:19.830349 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:19.830323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:43.036705 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.036626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:43.047164 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:36:43.047120    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:36:43.047164 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:36:43.047164 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:36:43.047164 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f" cmd=["/bin/sleep","15"]
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.047184    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" message="" err=<
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:36:43.047196    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerName="main" containerID="cri-o://d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"
Sep 18 19:36:43.047405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.047241    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerName="main" containerID="cri-o://d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f" gracePeriod=60
Sep 18 19:36:43.384720 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.384693    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:43.439196 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439196 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439195    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439369 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439369 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439242    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439369 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439299    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlgbz\" (UniqueName: \"kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439369 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm\") pod \"39a9537d-8161-4f3a-a662-e40a6ad648f9\" (UID: \"39a9537d-8161-4f3a-a662-e40a6ad648f9\") "
Sep 18 19:36:43.439567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:43.439567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439460    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home" (OuterVolumeSpecName: "home") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:43.439567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:43.439714 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439586    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.439714 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.439598    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.441667 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.441615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz" (OuterVolumeSpecName: "kube-api-access-zlgbz") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "kube-api-access-zlgbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:43.441667 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.441633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:43.441667 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.441642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm" (OuterVolumeSpecName: "dshm") pod "39a9537d-8161-4f3a-a662-e40a6ad648f9" (UID: "39a9537d-8161-4f3a-a662-e40a6ad648f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:43.540178 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.540150    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.540178 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.540179    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgbz\" (UniqueName: \"kubernetes.io/projected/39a9537d-8161-4f3a-a662-e40a6ad648f9-kube-api-access-zlgbz\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.540314 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.540191    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39a9537d-8161-4f3a-a662-e40a6ad648f9-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.540314 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.540200    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39a9537d-8161-4f3a-a662-e40a6ad648f9-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:36:43.933879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.933845    2582 generic.go:356] "Generic (PLEG): container finished" podID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerID="d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f" exitCode=0
Sep 18 19:36:43.934012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.933891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"
Sep 18 19:36:43.934012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.933931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" event={"ID":"39a9537d-8161-4f3a-a662-e40a6ad648f9","Type":"ContainerDied","Data":"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"}
Sep 18 19:36:43.934012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.933985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr" event={"ID":"39a9537d-8161-4f3a-a662-e40a6ad648f9","Type":"ContainerDied","Data":"28d4b79709e4331c7f60664a7bd9db393890c3b5c6d99d66e2a23fcd2b9ad956"}
Sep 18 19:36:43.934012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.934011    2582 scope.go:117] "RemoveContainer" containerID="d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"
Sep 18 19:36:43.948672 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.948651    2582 scope.go:117] "RemoveContainer" containerID="d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"
Sep 18 19:36:43.948980 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:36:43.948958    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f\": container with ID starting with d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f not found: ID does not exist" containerID="d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"
Sep 18 19:36:43.949066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.948987    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f"} err="failed to get container status \"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f\": rpc error: code = NotFound desc = could not find container \"d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f\": container with ID starting with d7baad38fafe6ada39849a504241dfc8c150b8d08da6f587aecad2652bc6b53f not found: ID does not exist"
Sep 18 19:36:43.956262 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.956240    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:43.961250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:43.961223    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vrfrr"]
Sep 18 19:36:45.895144 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:36:45.895108    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" path="/var/lib/kubelet/pods/39a9537d-8161-4f3a-a662-e40a6ad648f9/volumes"
Sep 18 19:37:03.709360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709324    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709627    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerName="main"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709638    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerName="main"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709660    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="storage-initializer"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="storage-initializer"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709679    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709684    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709746    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="666a78e2-53da-47bf-accf-7479f67fd260" containerName="main"
Sep 18 19:37:03.709797 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.709756    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a9537d-8161-4f3a-a662-e40a6ad648f9" containerName="main"
Sep 18 19:37:03.712708 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.712688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.716501 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.716482    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 19:37:03.716591 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.716534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-4mz4d\""
Sep 18 19:37:03.716591 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.716482    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 19:37:03.716904 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.716883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 19:37:03.725225 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.725205    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:37:03.779845 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.779822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.779955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.779850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.779955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.779900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.779955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.779932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hshmw\" (UniqueName: \"kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.780104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.779959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.780104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.780034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880638 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880638 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880638 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hshmw\" (UniqueName: \"kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880638 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.880959 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.881003 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.880980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.881048 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.881032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.883069 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.883040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.883178 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.883163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.890476 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.890452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hshmw\" (UniqueName: \"kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw\") pod \"fc-smoke-test-kserve-7f4687897-8fgk8\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:03.978612 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.978587    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:37:03.981850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.981834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:03.984309 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.984288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-65s7v\""
Sep 18 19:37:03.992444 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:03.992422    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:37:04.022274 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.022248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:04.083393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.083351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6qc\" (UniqueName: \"kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.083518 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.083423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.184413 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.184377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6qc\" (UniqueName: \"kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.184556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.184420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.186951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.186925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.193188 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.193169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6qc\" (UniqueName: \"kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.292768 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.292494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:04.294986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.294956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:37:04.296526 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:37:04.296507    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4e68a8_3843_464a_96b5_d1a8ec31fdb6.slice/crio-3b885c9ab3adbc961e9e4b357cdad0689a73fb2c03c0e12d879754facf7f4232 WatchSource:0}: Error finding container 3b885c9ab3adbc961e9e4b357cdad0689a73fb2c03c0e12d879754facf7f4232: Status 404 returned error can't find the container with id 3b885c9ab3adbc961e9e4b357cdad0689a73fb2c03c0e12d879754facf7f4232
Sep 18 19:37:04.573515 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:04.573491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:37:04.574984 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:37:04.574942    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2997fc_eeac_43c6_84cc_d38460ea8806.slice/crio-8ded1f82d1a8109ae456eec4aae83b6774e3b6b7ebd39eaa7f6419805d922917 WatchSource:0}: Error finding container 8ded1f82d1a8109ae456eec4aae83b6774e3b6b7ebd39eaa7f6419805d922917: Status 404 returned error can't find the container with id 8ded1f82d1a8109ae456eec4aae83b6774e3b6b7ebd39eaa7f6419805d922917
Sep 18 19:37:05.032414 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.032380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" event={"ID":"ec2997fc-eeac-43c6-84cc-d38460ea8806","Type":"ContainerStarted","Data":"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"}
Sep 18 19:37:05.032876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.032427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" event={"ID":"ec2997fc-eeac-43c6-84cc-d38460ea8806","Type":"ContainerStarted","Data":"8ded1f82d1a8109ae456eec4aae83b6774e3b6b7ebd39eaa7f6419805d922917"}
Sep 18 19:37:05.032876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.032463    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:37:05.034015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.033987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" event={"ID":"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6","Type":"ContainerStarted","Data":"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"}
Sep 18 19:37:05.034126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.034035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" event={"ID":"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6","Type":"ContainerStarted","Data":"3b885c9ab3adbc961e9e4b357cdad0689a73fb2c03c0e12d879754facf7f4232"}
Sep 18 19:37:05.053371 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.053320    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podStartSLOduration=2.053300898 podStartE2EDuration="2.053300898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:05.051879223 +0000 UTC m=+1357.289006210" watchObservedRunningTime="2026-09-18 19:37:05.053300898 +0000 UTC m=+1357.290427888"
Sep 18 19:37:05.074694 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:05.074648    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" podStartSLOduration=2.07463382 podStartE2EDuration="2.07463382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:05.07384002 +0000 UTC m=+1357.310967003" watchObservedRunningTime="2026-09-18 19:37:05.07463382 +0000 UTC m=+1357.311760800"
Sep 18 19:37:06.023194 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:06.023161    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:14.023120 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:14.023085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:14.035303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:14.035280    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:14.090030 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:14.090007    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:37:36.040404 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:37:36.040378    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:38:05.528001 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.527968    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:38:05.532962 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.532933    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:38:05.537792 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:38:05.537752    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:38:05.537792 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:38:05.537792 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:38:05.537792 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5" cmd=["/bin/sleep","15"]
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.537819    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" message="" err=<
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:38:05.537837    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerName="main" containerID="cri-o://a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"
Sep 18 19:38:05.538099 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.537877    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerName="main" containerID="cri-o://a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5" gracePeriod=60
Sep 18 19:38:05.888149 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.888124    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:38:05.938239 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938313    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hshmw\" (UniqueName: \"kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm\") pod \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\" (UID: \"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6\") "
Sep 18 19:38:05.938647 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938568    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:05.938703 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home" (OuterVolumeSpecName: "home") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:05.938799 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.938776    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:05.940773 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.940710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:38:05.940773 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.940761    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm" (OuterVolumeSpecName: "dshm") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:38:05.940901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:05.940824    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw" (OuterVolumeSpecName: "kube-api-access-hshmw") pod "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" (UID: "2e4e68a8-3843-464a-96b5-d1a8ec31fdb6"). InnerVolumeSpecName "kube-api-access-hshmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:38:06.039323 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039286    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.039323 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039317    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hshmw\" (UniqueName: \"kubernetes.io/projected/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-kube-api-access-hshmw\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.039323 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039326    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.039323 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039334    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.039589 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039341    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.039589 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.039368    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:06.040372 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.040342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:38:06.336583 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.336544    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerID="a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5" exitCode=0
Sep 18 19:38:06.336764 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.336589    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"
Sep 18 19:38:06.336764 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.336588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" event={"ID":"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6","Type":"ContainerDied","Data":"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"}
Sep 18 19:38:06.336764 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.336643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8" event={"ID":"2e4e68a8-3843-464a-96b5-d1a8ec31fdb6","Type":"ContainerDied","Data":"3b885c9ab3adbc961e9e4b357cdad0689a73fb2c03c0e12d879754facf7f4232"}
Sep 18 19:38:06.336764 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.336667    2582 scope.go:117] "RemoveContainer" containerID="a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"
Sep 18 19:38:06.346065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.346045    2582 scope.go:117] "RemoveContainer" containerID="a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"
Sep 18 19:38:06.346371 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:38:06.346345    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5\": container with ID starting with a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5 not found: ID does not exist" containerID="a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"
Sep 18 19:38:06.346419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.346381    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5"} err="failed to get container status \"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5\": rpc error: code = NotFound desc = could not find container \"a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5\": container with ID starting with a14d3c031eba78414304b7eaaf028978d7f663ab415a376607526c7c0bc71cc5 not found: ID does not exist"
Sep 18 19:38:06.375506 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.375478    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:38:06.377344 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:06.377320    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8fgk8"]
Sep 18 19:38:07.894880 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:07.894851    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" path="/var/lib/kubelet/pods/2e4e68a8-3843-464a-96b5-d1a8ec31fdb6/volumes"
Sep 18 19:38:16.041207 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:16.041130    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:38:20.553450 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:20.553411    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" containerID="cri-o://56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931" gracePeriod=45
Sep 18 19:38:26.041437 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:26.041403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:38:26.041849 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:26.041513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:38:36.041141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:36.041103    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:38:46.040990 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:46.040949    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:38:50.795980 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.795958    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:38:50.878543 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.878462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr6qc\" (UniqueName: \"kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc\") pod \"ec2997fc-eeac-43c6-84cc-d38460ea8806\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") "
Sep 18 19:38:50.878543 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.878519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs\") pod \"ec2997fc-eeac-43c6-84cc-d38460ea8806\" (UID: \"ec2997fc-eeac-43c6-84cc-d38460ea8806\") "
Sep 18 19:38:50.880784 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.880747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc" (OuterVolumeSpecName: "kube-api-access-lr6qc") pod "ec2997fc-eeac-43c6-84cc-d38460ea8806" (UID: "ec2997fc-eeac-43c6-84cc-d38460ea8806"). InnerVolumeSpecName "kube-api-access-lr6qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:38:50.880784 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.880753    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec2997fc-eeac-43c6-84cc-d38460ea8806" (UID: "ec2997fc-eeac-43c6-84cc-d38460ea8806"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:38:50.979529 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.979500    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2997fc-eeac-43c6-84cc-d38460ea8806-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:50.979529 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:50.979524    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6qc\" (UniqueName: \"kubernetes.io/projected/ec2997fc-eeac-43c6-84cc-d38460ea8806-kube-api-access-lr6qc\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:38:51.547024 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.546990    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerID="56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931" exitCode=0
Sep 18 19:38:51.547205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.547034    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"
Sep 18 19:38:51.547205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.547076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" event={"ID":"ec2997fc-eeac-43c6-84cc-d38460ea8806","Type":"ContainerDied","Data":"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"}
Sep 18 19:38:51.547205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.547129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww" event={"ID":"ec2997fc-eeac-43c6-84cc-d38460ea8806","Type":"ContainerDied","Data":"8ded1f82d1a8109ae456eec4aae83b6774e3b6b7ebd39eaa7f6419805d922917"}
Sep 18 19:38:51.547205 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.547150    2582 scope.go:117] "RemoveContainer" containerID="56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"
Sep 18 19:38:51.556151 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.556135    2582 scope.go:117] "RemoveContainer" containerID="56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"
Sep 18 19:38:51.556415 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:38:51.556395    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931\": container with ID starting with 56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931 not found: ID does not exist" containerID="56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"
Sep 18 19:38:51.556481 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.556423    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931"} err="failed to get container status \"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931\": rpc error: code = NotFound desc = could not find container \"56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931\": container with ID starting with 56f7a2264ec151f90dbdb94a7fe6543fd5743708ae99c72fd8927f3a66ff3931 not found: ID does not exist"
Sep 18 19:38:51.571674 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.571641    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:38:51.575663 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.575642    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-cc5ww"]
Sep 18 19:38:51.894603 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:38:51.894526    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" path="/var/lib/kubelet/pods/ec2997fc-eeac-43c6-84cc-d38460ea8806/volumes"
Sep 18 19:39:03.121539 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.121503    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:03.121948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.121903    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main"
Sep 18 19:39:03.121948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.121921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main"
Sep 18 19:39:03.122173 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.121952    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerName="main"
Sep 18 19:39:03.122173 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.121959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerName="main"
Sep 18 19:39:03.122173 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.122020    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4e68a8-3843-464a-96b5-d1a8ec31fdb6" containerName="main"
Sep 18 19:39:03.122173 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.122033    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2997fc-eeac-43c6-84cc-d38460ea8806" containerName="main"
Sep 18 19:39:03.126280 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.126262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.129115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.129091    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 18 19:39:03.129881 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.129863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 18 19:39:03.129995 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.129975    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-4vtxc\""
Sep 18 19:39:03.130061 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.129977    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 18 19:39:03.146311 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.146289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:03.279900 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.279870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.280065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.279906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.280065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.279924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.280065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.279955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.280065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.280021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.280065 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.280060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftj8z\" (UniqueName: \"kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381272 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381272 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381272 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftj8z\" (UniqueName: \"kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381693 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381693 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.381795 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.381769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.383782 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.383754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.384093 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.384072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.385720 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.385698    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:39:03.389537 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.389518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.392161 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.392138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-hfhjh\""
Sep 18 19:39:03.393056 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.393040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftj8z\" (UniqueName: \"kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z\") pod \"fc-concurrency-test-kserve-b5d44467c-x9r99\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.397287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.397268    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:39:03.436895 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.436873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:03.482634 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.482601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8ss\" (UniqueName: \"kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.482811 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.482660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.584088 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.584060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.584221 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.584131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8ss\" (UniqueName: \"kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.586685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.586658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.593459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.593437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8ss\" (UniqueName: \"kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.712911 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.712833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:03.730994 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:03.730968    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:03.731823 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:39:03.731792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57728263_6978_4369_93d0_a6b2fca453f5.slice/crio-6422f3a999973edb79dfebdb591105e3fd1c6545bf594d55fd0ffc1fab7b6a3b WatchSource:0}: Error finding container 6422f3a999973edb79dfebdb591105e3fd1c6545bf594d55fd0ffc1fab7b6a3b: Status 404 returned error can't find the container with id 6422f3a999973edb79dfebdb591105e3fd1c6545bf594d55fd0ffc1fab7b6a3b
Sep 18 19:39:04.047431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.047387    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:39:04.051666 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:39:04.051637    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a11390d_b2cd_41cd_9f87_af6eb8b83fed.slice/crio-fb0e385bc66798a500e65a56077fa34ed748cd135e65921ed1cf7c0edd972e3e WatchSource:0}: Error finding container fb0e385bc66798a500e65a56077fa34ed748cd135e65921ed1cf7c0edd972e3e: Status 404 returned error can't find the container with id fb0e385bc66798a500e65a56077fa34ed748cd135e65921ed1cf7c0edd972e3e
Sep 18 19:39:04.608809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.608769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" event={"ID":"57728263-6978-4369-93d0-a6b2fca453f5","Type":"ContainerStarted","Data":"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"}
Sep 18 19:39:04.609266 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.608831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" event={"ID":"57728263-6978-4369-93d0-a6b2fca453f5","Type":"ContainerStarted","Data":"6422f3a999973edb79dfebdb591105e3fd1c6545bf594d55fd0ffc1fab7b6a3b"}
Sep 18 19:39:04.610438 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.610415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" event={"ID":"2a11390d-b2cd-41cd-9f87-af6eb8b83fed","Type":"ContainerStarted","Data":"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"}
Sep 18 19:39:04.610560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.610452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" event={"ID":"2a11390d-b2cd-41cd-9f87-af6eb8b83fed","Type":"ContainerStarted","Data":"fb0e385bc66798a500e65a56077fa34ed748cd135e65921ed1cf7c0edd972e3e"}
Sep 18 19:39:04.610560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.610545    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:04.667519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.667468    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" podStartSLOduration=1.667453359 podStartE2EDuration="1.667453359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:04.665020428 +0000 UTC m=+1476.902147599" watchObservedRunningTime="2026-09-18 19:39:04.667453359 +0000 UTC m=+1476.904580345"
Sep 18 19:39:04.701862 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:04.701815    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podStartSLOduration=1.70180241 podStartE2EDuration="1.70180241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:04.699869792 +0000 UTC m=+1476.936996778" watchObservedRunningTime="2026-09-18 19:39:04.70180241 +0000 UTC m=+1476.938929396"
Sep 18 19:39:05.437882 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:05.437846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:13.437676 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:13.437639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:13.450190 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:13.450167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:13.664527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:13.664491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:29.865544 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:29.865515    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:39:29.868978 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:29.868953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:39:29.873028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:29.873008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:39:29.875855 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:29.875837    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:39:35.616744 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:35.616700    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:39:42.129556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.129469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:39:42.134497 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.134472    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:42.143908 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:39:42.143870    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:39:42.143908 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:42.143908 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:42.143908 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737" cmd=["/bin/sleep","15"]
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.143923    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" message="" err=<
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:39:42.143939    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" podUID="57728263-6978-4369-93d0-a6b2fca453f5" containerName="main" containerID="cri-o://f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"
Sep 18 19:39:42.144212 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.143985    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" podUID="57728263-6978-4369-93d0-a6b2fca453f5" containerName="main" containerID="cri-o://f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737" gracePeriod=60
Sep 18 19:39:42.506201 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.506171    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:42.580747 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580690    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.580930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580775    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.580930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580823    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.580930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580855    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftj8z\" (UniqueName: \"kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.580930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580919    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm\") pod \"57728263-6978-4369-93d0-a6b2fca453f5\" (UID: \"57728263-6978-4369-93d0-a6b2fca453f5\") "
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.580982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home" (OuterVolumeSpecName: "home") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.581051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.581145    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.581227    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.581245    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.581287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.581258    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.583503 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.583469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm" (OuterVolumeSpecName: "dshm") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:42.583503 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.583478    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z" (OuterVolumeSpecName: "kube-api-access-ftj8z") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "kube-api-access-ftj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:42.583689 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.583506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57728263-6978-4369-93d0-a6b2fca453f5" (UID: "57728263-6978-4369-93d0-a6b2fca453f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:42.682293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.682212    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57728263-6978-4369-93d0-a6b2fca453f5-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.682293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.682243    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57728263-6978-4369-93d0-a6b2fca453f5-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.682293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.682261    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftj8z\" (UniqueName: \"kubernetes.io/projected/57728263-6978-4369-93d0-a6b2fca453f5-kube-api-access-ftj8z\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:39:42.809326 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.809296    2582 generic.go:356] "Generic (PLEG): container finished" podID="57728263-6978-4369-93d0-a6b2fca453f5" containerID="f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737" exitCode=0
Sep 18 19:39:42.809478 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.809340    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"
Sep 18 19:39:42.809478 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.809373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" event={"ID":"57728263-6978-4369-93d0-a6b2fca453f5","Type":"ContainerDied","Data":"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"}
Sep 18 19:39:42.809478 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.809423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99" event={"ID":"57728263-6978-4369-93d0-a6b2fca453f5","Type":"ContainerDied","Data":"6422f3a999973edb79dfebdb591105e3fd1c6545bf594d55fd0ffc1fab7b6a3b"}
Sep 18 19:39:42.809478 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.809446    2582 scope.go:117] "RemoveContainer" containerID="f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"
Sep 18 19:39:42.818722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.818707    2582 scope.go:117] "RemoveContainer" containerID="f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"
Sep 18 19:39:42.819055 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:39:42.819013    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737\": container with ID starting with f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737 not found: ID does not exist" containerID="f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"
Sep 18 19:39:42.819055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.819042    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737"} err="failed to get container status \"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737\": rpc error: code = NotFound desc = could not find container \"f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737\": container with ID starting with f0123e110345ff3c5413596ee4bfa78d3d4ff20593e58892d4f50e2ee16e4737 not found: ID does not exist"
Sep 18 19:39:42.831741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.831702    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:42.836911 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:42.836890    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-x9r99"]
Sep 18 19:39:43.895737 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:43.895691    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57728263-6978-4369-93d0-a6b2fca453f5" path="/var/lib/kubelet/pods/57728263-6978-4369-93d0-a6b2fca453f5/volumes"
Sep 18 19:39:45.617015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:45.616983    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:39:55.617374 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:55.617342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:39:57.154466 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:39:57.154428    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" containerID="cri-o://edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5" gracePeriod=45
Sep 18 19:40:05.617063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:05.617025    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:40:05.617612 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:05.617140    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:40:15.617099 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:15.617060    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:40:21.285299 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.285268    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:40:21.285653 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.285550    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57728263-6978-4369-93d0-a6b2fca453f5" containerName="main"
Sep 18 19:40:21.285653 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.285560    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57728263-6978-4369-93d0-a6b2fca453f5" containerName="main"
Sep 18 19:40:21.285653 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.285619    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57728263-6978-4369-93d0-a6b2fca453f5" containerName="main"
Sep 18 19:40:21.288219 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.288200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.293184 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.293161    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 19:40:21.293184 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.293177    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 19:40:21.293364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.293161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-r66hn\""
Sep 18 19:40:21.293364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.293161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 19:40:21.301915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.301894    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:40:21.469972 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.469938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.470169 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.469983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.470169 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.470051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.470169 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.470151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.470284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.470192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqcn\" (UniqueName: \"kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.470284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.470218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571502 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571502 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqcn\" (UniqueName: \"kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571995 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.571995 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.571989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.574134 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.574106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.574287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.574267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.586321 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.586296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqcn\" (UniqueName: \"kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.600005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.599987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:21.879004 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.878669    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:40:21.882058 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:40:21.882029    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cc0402_390f_40e8_9f54_fc6b75f95aaf.slice/crio-c5e5e4fa7131b2d5fc5fe7f963412c6e6c731f941f84e3434199c302e7079c86 WatchSource:0}: Error finding container c5e5e4fa7131b2d5fc5fe7f963412c6e6c731f941f84e3434199c302e7079c86: Status 404 returned error can't find the container with id c5e5e4fa7131b2d5fc5fe7f963412c6e6c731f941f84e3434199c302e7079c86
Sep 18 19:40:21.998186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.998152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" event={"ID":"32cc0402-390f-40e8-9f54-fc6b75f95aaf","Type":"ContainerStarted","Data":"41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65"}
Sep 18 19:40:21.998341 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:21.998202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" event={"ID":"32cc0402-390f-40e8-9f54-fc6b75f95aaf","Type":"ContainerStarted","Data":"c5e5e4fa7131b2d5fc5fe7f963412c6e6c731f941f84e3434199c302e7079c86"}
Sep 18 19:40:22.020057 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:22.020012    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" podStartSLOduration=1.019998678 podStartE2EDuration="1.019998678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:22.018052732 +0000 UTC m=+1554.255179717" watchObservedRunningTime="2026-09-18 19:40:22.019998678 +0000 UTC m=+1554.257125672"
Sep 18 19:40:22.600746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:22.600696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:25.617193 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:25.617156    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:40:27.403222 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.403198    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:40:27.413120 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.413065    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs\") pod \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") "
Sep 18 19:40:27.413229 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.413163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z8ss\" (UniqueName: \"kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss\") pod \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\" (UID: \"2a11390d-b2cd-41cd-9f87-af6eb8b83fed\") "
Sep 18 19:40:27.415124 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.415092    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a11390d-b2cd-41cd-9f87-af6eb8b83fed" (UID: "2a11390d-b2cd-41cd-9f87-af6eb8b83fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:27.415223 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.415143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss" (OuterVolumeSpecName: "kube-api-access-5z8ss") pod "2a11390d-b2cd-41cd-9f87-af6eb8b83fed" (UID: "2a11390d-b2cd-41cd-9f87-af6eb8b83fed"). InnerVolumeSpecName "kube-api-access-5z8ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:27.514015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.513986    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8ss\" (UniqueName: \"kubernetes.io/projected/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-kube-api-access-5z8ss\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:40:27.514015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:27.514011    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11390d-b2cd-41cd-9f87-af6eb8b83fed-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:40:28.027426 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.027391    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerID="edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5" exitCode=0
Sep 18 19:40:28.027566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.027437    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"
Sep 18 19:40:28.027566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.027481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" event={"ID":"2a11390d-b2cd-41cd-9f87-af6eb8b83fed","Type":"ContainerDied","Data":"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"}
Sep 18 19:40:28.027566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.027533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd" event={"ID":"2a11390d-b2cd-41cd-9f87-af6eb8b83fed","Type":"ContainerDied","Data":"fb0e385bc66798a500e65a56077fa34ed748cd135e65921ed1cf7c0edd972e3e"}
Sep 18 19:40:28.027566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.027561    2582 scope.go:117] "RemoveContainer" containerID="edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"
Sep 18 19:40:28.036037 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.036017    2582 scope.go:117] "RemoveContainer" containerID="edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"
Sep 18 19:40:28.036312 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:40:28.036283    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5\": container with ID starting with edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5 not found: ID does not exist" containerID="edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"
Sep 18 19:40:28.036406 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.036313    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5"} err="failed to get container status \"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5\": rpc error: code = NotFound desc = could not find container \"edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5\": container with ID starting with edb990bd52710be04e96884f2fb468db5448edeb8f8f7db774aee3edd067faf5 not found: ID does not exist"
Sep 18 19:40:28.048045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.048025    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:40:28.051467 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:28.051446    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jfmfd"]
Sep 18 19:40:29.895303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:29.895273    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" path="/var/lib/kubelet/pods/2a11390d-b2cd-41cd-9f87-af6eb8b83fed/volumes"
Sep 18 19:40:31.601154 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:31.601112    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:31.613862 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:31.613830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:40:32.058500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:40:32.058469    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:41:15.199221 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.199141    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:15.199616 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.199490    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main"
Sep 18 19:41:15.199616 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.199503    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main"
Sep 18 19:41:15.199616 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.199559    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a11390d-b2cd-41cd-9f87-af6eb8b83fed" containerName="main"
Sep 18 19:41:15.202590 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.202574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.207913 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.207888    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-4wfx7\""
Sep 18 19:41:15.208058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.207919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 18 19:41:15.208058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.207935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 18 19:41:15.209858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.209841    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 18 19:41:15.220200 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.220179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:15.291247 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.291247 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.291469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66p2d\" (UniqueName: \"kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.291469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.291469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.291469 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.291415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392543 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392543 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.392891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p2d\" (UniqueName: \"kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.393123 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.392951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.393123 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.393105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.393244 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.393122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.395045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.395022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.395312 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.395295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.404519 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.404496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66p2d\" (UniqueName: \"kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.512201 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.512158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:15.817915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:15.817875    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:15.819383 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:41:15.819358    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf671c5fe_2cc7_4e58_a7f3_6ca2ee802148.slice/crio-3bfdee6e1727317b5ead847ef723c6c8821590d12c6988158235a095c805e439 WatchSource:0}: Error finding container 3bfdee6e1727317b5ead847ef723c6c8821590d12c6988158235a095c805e439: Status 404 returned error can't find the container with id 3bfdee6e1727317b5ead847ef723c6c8821590d12c6988158235a095c805e439
Sep 18 19:41:16.263637 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:16.263598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" event={"ID":"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148","Type":"ContainerStarted","Data":"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"}
Sep 18 19:41:16.264120 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:16.263669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" event={"ID":"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148","Type":"ContainerStarted","Data":"3bfdee6e1727317b5ead847ef723c6c8821590d12c6988158235a095c805e439"}
Sep 18 19:41:16.291370 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:16.291303    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" podStartSLOduration=1.291288834 podStartE2EDuration="1.291288834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:16.291204821 +0000 UTC m=+1608.528331808" watchObservedRunningTime="2026-09-18 19:41:16.291288834 +0000 UTC m=+1608.528415820"
Sep 18 19:41:16.513305 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:16.513273    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:21.817971 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:21.817935    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:21.827195 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:21.827157    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:41:21.827195 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:21.827195 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:21.827195 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce" cmd=["/bin/sleep","15"]
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:21.827208    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" message="" err=<
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:21.827219    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerName="main" containerID="cri-o://ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"
Sep 18 19:41:21.827429 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:21.827257    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerName="main" containerID="cri-o://ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce" gracePeriod=60
Sep 18 19:41:22.066939 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.066915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:22.149046 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.148982    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149046 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149232 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149054    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66p2d\" (UniqueName: \"kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149232 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149088    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149232 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149232 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149158    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir\") pod \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\" (UID: \"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148\") "
Sep 18 19:41:22.149439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache" (OuterVolumeSpecName: "model-cache") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:22.149439 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149381    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home" (OuterVolumeSpecName: "home") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:22.149617 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.149487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:22.151654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.151622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:22.151654 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.151620    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm" (OuterVolumeSpecName: "dshm") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:22.151858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.151770    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d" (OuterVolumeSpecName: "kube-api-access-66p2d") pod "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" (UID: "f671c5fe-2cc7-4e58-a7f3-6ca2ee802148"). InnerVolumeSpecName "kube-api-access-66p2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:22.249696 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249663    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.249696 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249690    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.249696 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249701    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66p2d\" (UniqueName: \"kubernetes.io/projected/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-kube-api-access-66p2d\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.249922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249710    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.249922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249722    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.249922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.249749    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:22.292623 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.292586    2582 generic.go:356] "Generic (PLEG): container finished" podID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerID="ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce" exitCode=0
Sep 18 19:41:22.292807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.292643    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"
Sep 18 19:41:22.292807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.292669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" event={"ID":"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148","Type":"ContainerDied","Data":"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"}
Sep 18 19:41:22.292807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.292739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd" event={"ID":"f671c5fe-2cc7-4e58-a7f3-6ca2ee802148","Type":"ContainerDied","Data":"3bfdee6e1727317b5ead847ef723c6c8821590d12c6988158235a095c805e439"}
Sep 18 19:41:22.292807 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.292762    2582 scope.go:117] "RemoveContainer" containerID="ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"
Sep 18 19:41:22.303061 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.303046    2582 scope.go:117] "RemoveContainer" containerID="ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"
Sep 18 19:41:22.303327 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:22.303306    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce\": container with ID starting with ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce not found: ID does not exist" containerID="ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"
Sep 18 19:41:22.303379 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.303335    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce"} err="failed to get container status \"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce\": rpc error: code = NotFound desc = could not find container \"ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce\": container with ID starting with ba8b18e196730f4744e38c42b617a460a54b7076eccb539d3b0329306b9ee0ce not found: ID does not exist"
Sep 18 19:41:22.315924 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.315903    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:22.320103 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:22.320083    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-54fhd"]
Sep 18 19:41:23.898366 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:23.898330    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" path="/var/lib/kubelet/pods/f671c5fe-2cc7-4e58-a7f3-6ca2ee802148/volumes"
Sep 18 19:41:49.567247 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.567202    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:49.567863 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.567597    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerName="main"
Sep 18 19:41:49.567863 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.567609    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerName="main"
Sep 18 19:41:49.567863 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.567672    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f671c5fe-2cc7-4e58-a7f3-6ca2ee802148" containerName="main"
Sep 18 19:41:49.570481 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.570462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.573841 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.573817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 18 19:41:49.573841 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.573832    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-2lxqf\""
Sep 18 19:41:49.574028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.573843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 19:41:49.574028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.573817    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 19:41:49.582379 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.582357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:49.642035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.641996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.642035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.642035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlm7\" (UniqueName: \"kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.642283 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.642069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.642283 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.642094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.642283 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.642156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.642283 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.642181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743087 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743087 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlm7\" (UniqueName: \"kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.743643 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.743604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.745608 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.745587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.745891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.745874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.752292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.752256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlm7\" (UniqueName: \"kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:49.881465 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:49.881372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:50.155632 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:50.155557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:50.159066 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:41:50.159036    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae122af6_7f55_4cd6_a3ac_338a3e0b9af5.slice/crio-e980d8891357d101a8afd02c278ed88d505f6a0da5355737a46bfe220d8bbfde WatchSource:0}: Error finding container e980d8891357d101a8afd02c278ed88d505f6a0da5355737a46bfe220d8bbfde: Status 404 returned error can't find the container with id e980d8891357d101a8afd02c278ed88d505f6a0da5355737a46bfe220d8bbfde
Sep 18 19:41:50.428078 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:50.427995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" event={"ID":"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5","Type":"ContainerStarted","Data":"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"}
Sep 18 19:41:50.428078 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:50.428071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" event={"ID":"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5","Type":"ContainerStarted","Data":"e980d8891357d101a8afd02c278ed88d505f6a0da5355737a46bfe220d8bbfde"}
Sep 18 19:41:50.446534 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:50.446479    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" podStartSLOduration=1.446464776 podStartE2EDuration="1.446464776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:50.444667397 +0000 UTC m=+1642.681794384" watchObservedRunningTime="2026-09-18 19:41:50.446464776 +0000 UTC m=+1642.683591761"
Sep 18 19:41:50.882170 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:50.882135    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:52.192308 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.192275    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:41:52.201259 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:52.201217    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:41:52.201259 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.201259 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.201259 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65" cmd=["/bin/sleep","15"]
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.201285    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" message="" err=<
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:52.201299    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerName="main" containerID="cri-o://41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65"
Sep 18 19:41:52.201541 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.201348    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerName="main" containerID="cri-o://41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65" gracePeriod=60
Sep 18 19:41:52.441809 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.441779    2582 generic.go:356] "Generic (PLEG): container finished" podID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerID="41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65" exitCode=0
Sep 18 19:41:52.441996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.441866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" event={"ID":"32cc0402-390f-40e8-9f54-fc6b75f95aaf","Type":"ContainerDied","Data":"41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65"}
Sep 18 19:41:52.537746 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.537705    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:41:52.567322 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567335    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567359    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567399    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567444    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567609 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567510    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hqcn\" (UniqueName: \"kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn\") pod \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\" (UID: \"32cc0402-390f-40e8-9f54-fc6b75f95aaf\") "
Sep 18 19:41:52.567652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.567706 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home" (OuterVolumeSpecName: "home") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.567788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.567788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567782    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.567876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567795    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.567876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.567804    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.569741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.569705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn" (OuterVolumeSpecName: "kube-api-access-2hqcn") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "kube-api-access-2hqcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:52.569844 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.569752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm" (OuterVolumeSpecName: "dshm") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:52.569844 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.569759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32cc0402-390f-40e8-9f54-fc6b75f95aaf" (UID: "32cc0402-390f-40e8-9f54-fc6b75f95aaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:52.668213 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.668185    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqcn\" (UniqueName: \"kubernetes.io/projected/32cc0402-390f-40e8-9f54-fc6b75f95aaf-kube-api-access-2hqcn\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.668213 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.668211    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cc0402-390f-40e8-9f54-fc6b75f95aaf-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.668213 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.668221    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cc0402-390f-40e8-9f54-fc6b75f95aaf-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:52.709348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:52.709317    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:53.448577 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.448545    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"
Sep 18 19:41:53.448577 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.448569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj" event={"ID":"32cc0402-390f-40e8-9f54-fc6b75f95aaf","Type":"ContainerDied","Data":"c5e5e4fa7131b2d5fc5fe7f963412c6e6c731f941f84e3434199c302e7079c86"}
Sep 18 19:41:53.449135 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.448651    2582 scope.go:117] "RemoveContainer" containerID="41998de816f26bc3956df3ae478f00f0af3dc62d29190c37cd07e9699d515f65"
Sep 18 19:41:53.459025 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:53.458994    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:41:53.459025 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:53.459025 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:53.459025 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd" cmd=["/bin/sleep","15"]
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.459047    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" message="" err=<
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:53.459064    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerName="main" containerID="cri-o://3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"
Sep 18 19:41:53.459266 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.459107    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerName="main" containerID="cri-o://3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd" gracePeriod=60
Sep 18 19:41:53.526752 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.526712    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:41:53.532956 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.532934    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444lg4dj"]
Sep 18 19:41:53.702580 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.702520    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:53.775598 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775573    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.775796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.775796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.775796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdlm7\" (UniqueName: \"kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.775796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775717    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.775796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs\") pod \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\" (UID: \"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5\") "
Sep 18 19:41:53.776070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:53.776070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:53.776070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.775989    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.776070 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.776012    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home" (OuterVolumeSpecName: "home") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:53.778009 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.777985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7" (OuterVolumeSpecName: "kube-api-access-kdlm7") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "kube-api-access-kdlm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:53.778101 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.778008    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:53.778101 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.778060    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm" (OuterVolumeSpecName: "dshm") pod "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" (UID: "ae122af6-7f55-4cd6-a3ac-338a3e0b9af5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:41:53.876587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.876557    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.876587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.876586    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.876772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.876614    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.876772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.876623    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlm7\" (UniqueName: \"kubernetes.io/projected/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-kube-api-access-kdlm7\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.876772 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.876631    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:41:53.895450 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:53.895425    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" path="/var/lib/kubelet/pods/32cc0402-390f-40e8-9f54-fc6b75f95aaf/volumes"
Sep 18 19:41:54.453811 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.453774    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerID="3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd" exitCode=0
Sep 18 19:41:54.454297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.453824    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"
Sep 18 19:41:54.454297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.453869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" event={"ID":"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5","Type":"ContainerDied","Data":"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"}
Sep 18 19:41:54.454297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.453914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2" event={"ID":"ae122af6-7f55-4cd6-a3ac-338a3e0b9af5","Type":"ContainerDied","Data":"e980d8891357d101a8afd02c278ed88d505f6a0da5355737a46bfe220d8bbfde"}
Sep 18 19:41:54.454297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.453943    2582 scope.go:117] "RemoveContainer" containerID="3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"
Sep 18 19:41:54.462719 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.462703    2582 scope.go:117] "RemoveContainer" containerID="3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"
Sep 18 19:41:54.462981 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:41:54.462965    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd\": container with ID starting with 3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd not found: ID does not exist" containerID="3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"
Sep 18 19:41:54.463034 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.462986    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd"} err="failed to get container status \"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd\": rpc error: code = NotFound desc = could not find container \"3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd\": container with ID starting with 3ee285b7420ed98fa82aef79a48a90e0e363fcdfb72bb99cbead05473810d5dd not found: ID does not exist"
Sep 18 19:41:54.471512 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.471491    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:54.474387 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:54.474368    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf4k6c2"]
Sep 18 19:41:55.895502 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:41:55.895461    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" path="/var/lib/kubelet/pods/ae122af6-7f55-4cd6-a3ac-338a3e0b9af5/volumes"
Sep 18 19:42:02.325466 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325431    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325714    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerName="main"
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325738    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerName="main"
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325750    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerName="main"
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325757    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerName="main"
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325818    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cc0402-390f-40e8-9f54-fc6b75f95aaf" containerName="main"
Sep 18 19:42:02.325878 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.325830    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae122af6-7f55-4cd6-a3ac-338a3e0b9af5" containerName="main"
Sep 18 19:42:02.330113 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.330095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.336692 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.336314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 18 19:42:02.336692 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.336469    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:42:02.336692 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.336573    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 19:42:02.336875 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.336806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-phfwf\""
Sep 18 19:42:02.336875 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.336842    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 19:42:02.339440 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.339423    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 19:42:02.383259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.383232    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:42:02.432069 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432189 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.432405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.432223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94dd\" (UniqueName: \"kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533271 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533495 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94dd\" (UniqueName: \"kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533623 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533679 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.533835 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.533812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.534056 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.534039    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.535661 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.535642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.536047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.536028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.551104 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.551085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94dd\" (UniqueName: \"kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd\") pod \"auth-enabled-test-kserve-7544c47947-65rs6\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.640156 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.640078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:02.669267 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.669237    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:42:02.672801 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.672779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.676269 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.676251    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-nrn2c\""
Sep 18 19:42:02.701397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.701368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:42:02.735769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.735182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf9q\" (UniqueName: \"kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.735769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.735285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.836384 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.836348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf9q\" (UniqueName: \"kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.836545 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.836395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.839001 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.838979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.846670 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.846648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf9q\" (UniqueName: \"kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:02.927095 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.927027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:42:02.929242 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:42:02.929213    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805add19_4623_4b7d_be58_c654ff2030c8.slice/crio-66eea907e44b0487d18ceded201483412742c1250fe47205b13be5e52be8b9a8 WatchSource:0}: Error finding container 66eea907e44b0487d18ceded201483412742c1250fe47205b13be5e52be8b9a8: Status 404 returned error can't find the container with id 66eea907e44b0487d18ceded201483412742c1250fe47205b13be5e52be8b9a8
Sep 18 19:42:02.983870 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:02.983848    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:03.267393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.267362    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:42:03.268538 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:42:03.268504    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3adf1b_bbee_4266_bef5_3ac5fa77611e.slice/crio-b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6 WatchSource:0}: Error finding container b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6: Status 404 returned error can't find the container with id b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6
Sep 18 19:42:03.504782 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.504740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" event={"ID":"ea3adf1b-bbee-4266-bef5-3ac5fa77611e","Type":"ContainerStarted","Data":"5318d64137f3c2c24ec402a6078a1b705362fd232ca2abd56c2b80aecf86c32e"}
Sep 18 19:42:03.505234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.504805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" event={"ID":"ea3adf1b-bbee-4266-bef5-3ac5fa77611e","Type":"ContainerStarted","Data":"b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6"}
Sep 18 19:42:03.505234 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.504921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:03.506454 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.506423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerStarted","Data":"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"}
Sep 18 19:42:03.506566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.506478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerStarted","Data":"66eea907e44b0487d18ceded201483412742c1250fe47205b13be5e52be8b9a8"}
Sep 18 19:42:03.529244 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:03.529122    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podStartSLOduration=1.529103101 podStartE2EDuration="1.529103101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:03.52822253 +0000 UTC m=+1655.765349516" watchObservedRunningTime="2026-09-18 19:42:03.529103101 +0000 UTC m=+1655.766230089"
Sep 18 19:42:05.518944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:05.518852    2582 generic.go:356] "Generic (PLEG): container finished" podID="805add19-4623-4b7d-be58-c654ff2030c8" containerID="85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8" exitCode=0
Sep 18 19:42:05.519424 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:05.518930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerDied","Data":"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"}
Sep 18 19:42:06.526712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:06.526676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerStarted","Data":"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"}
Sep 18 19:42:06.640840 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:06.640801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:12.641123 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:12.641086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:42:12.642650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:12.642621    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:42:22.640758 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:22.640688    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:42:32.640774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:32.640701    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:42:34.513885 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:34.513855    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:42:34.534751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:34.534668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podStartSLOduration=30.306945856 podStartE2EDuration="32.534645648s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.227699792s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:06.558419635 +0000 UTC m=+1658.795546621" watchObservedRunningTime="2026-09-18 19:42:34.534645648 +0000 UTC m=+1686.771772635"
Sep 18 19:42:42.641029 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:42.640933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:42:52.641472 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:42:52.640996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:02.641158 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:02.641106    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:12.641093 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:12.641050    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:15.632930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.632896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:15.635386 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.635363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.638749 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.638702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-xlhct\""
Sep 18 19:43:15.638884 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.638761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:43:15.638984 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.638947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 19:43:15.640466 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.640445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 19:43:15.649822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.649760    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:15.744550 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.744475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.744861 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.744598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.744861 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.744776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.744861 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.744840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.845912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.845876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.845912 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.845915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.846186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.845937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.846186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.845966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.846361 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.846334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.846425 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.846374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.846582 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.846564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.848668 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.848648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:15.949615 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:15.949534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:16.225834 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:16.225807    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:16.226864 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:43:16.226838    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ca41ff_2ff9_4359_8c04_47470493924f.slice/crio-b8c1236736772a5091db51b8a311746fe2e4b62c5ee8d26131aa9b5c8d7c8421 WatchSource:0}: Error finding container b8c1236736772a5091db51b8a311746fe2e4b62c5ee8d26131aa9b5c8d7c8421: Status 404 returned error can't find the container with id b8c1236736772a5091db51b8a311746fe2e4b62c5ee8d26131aa9b5c8d7c8421
Sep 18 19:43:16.892688 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:16.892648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerStarted","Data":"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"}
Sep 18 19:43:16.893201 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:16.892750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerStarted","Data":"b8c1236736772a5091db51b8a311746fe2e4b62c5ee8d26131aa9b5c8d7c8421"}
Sep 18 19:43:18.907575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:18.907481    2582 generic.go:356] "Generic (PLEG): container finished" podID="18ca41ff-2ff9-4359-8c04-47470493924f" containerID="b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08" exitCode=0
Sep 18 19:43:18.907983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:18.907568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerDied","Data":"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"}
Sep 18 19:43:19.914835 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:19.914796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerStarted","Data":"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"}
Sep 18 19:43:19.915337 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:19.915023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:19.947697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:19.947651    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" podStartSLOduration=2.635202492 podStartE2EDuration="4.947637181s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.312434689s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:19.943942281 +0000 UTC m=+1732.181069280" watchObservedRunningTime="2026-09-18 19:43:19.947637181 +0000 UTC m=+1732.184764199"
Sep 18 19:43:22.641287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:22.641233    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:32.640982 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:32.640929    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:42.641331 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:42.641270    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:50.933110 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:50.933074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:52.033282 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:52.033257    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm_18ca41ff-2ff9-4359-8c04-47470493924f/main/0.log"
Sep 18 19:43:52.641305 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:52.641260    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:43:52.764170 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:52.764135    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:52.764368 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:52.764345    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="main" containerID="cri-o://a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc" gracePeriod=30
Sep 18 19:43:53.417576 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.417551    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:53.492334 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.492223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert\") pod \"18ca41ff-2ff9-4359-8c04-47470493924f\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") "
Sep 18 19:43:53.492334 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.492319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs\") pod \"18ca41ff-2ff9-4359-8c04-47470493924f\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") "
Sep 18 19:43:53.492572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.492352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir\") pod \"18ca41ff-2ff9-4359-8c04-47470493924f\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") "
Sep 18 19:43:53.492572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.492393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location\") pod \"18ca41ff-2ff9-4359-8c04-47470493924f\" (UID: \"18ca41ff-2ff9-4359-8c04-47470493924f\") "
Sep 18 19:43:53.492680 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.492616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18ca41ff-2ff9-4359-8c04-47470493924f" (UID: "18ca41ff-2ff9-4359-8c04-47470493924f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:53.493557 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.493527    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18ca41ff-2ff9-4359-8c04-47470493924f" (UID: "18ca41ff-2ff9-4359-8c04-47470493924f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.494934 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.494909    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18ca41ff-2ff9-4359-8c04-47470493924f" (UID: "18ca41ff-2ff9-4359-8c04-47470493924f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:53.513673 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.513644    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18ca41ff-2ff9-4359-8c04-47470493924f" (UID: "18ca41ff-2ff9-4359-8c04-47470493924f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:53.594047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.594005    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca41ff-2ff9-4359-8c04-47470493924f-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.594047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.594055    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.594307 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.594065    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ca41ff-2ff9-4359-8c04-47470493924f-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:43:53.594307 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:53.594075    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ca41ff-2ff9-4359-8c04-47470493924f-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:43:54.100196 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.100146    2582 generic.go:356] "Generic (PLEG): container finished" podID="18ca41ff-2ff9-4359-8c04-47470493924f" containerID="a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc" exitCode=0
Sep 18 19:43:54.100401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.100205    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"
Sep 18 19:43:54.100401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.100267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerDied","Data":"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"}
Sep 18 19:43:54.100401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.100310    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm" event={"ID":"18ca41ff-2ff9-4359-8c04-47470493924f","Type":"ContainerDied","Data":"b8c1236736772a5091db51b8a311746fe2e4b62c5ee8d26131aa9b5c8d7c8421"}
Sep 18 19:43:54.100401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.100332    2582 scope.go:117] "RemoveContainer" containerID="a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"
Sep 18 19:43:54.109237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.109223    2582 scope.go:117] "RemoveContainer" containerID="b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"
Sep 18 19:43:54.117239 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.117216    2582 scope.go:117] "RemoveContainer" containerID="a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"
Sep 18 19:43:54.117516 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:43:54.117498    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc\": container with ID starting with a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc not found: ID does not exist" containerID="a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"
Sep 18 19:43:54.117562 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.117525    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc"} err="failed to get container status \"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc\": rpc error: code = NotFound desc = could not find container \"a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc\": container with ID starting with a91576d6b765653de72e359f81689c729cd46224a67e8f737c7793796b6c70cc not found: ID does not exist"
Sep 18 19:43:54.117562 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.117543    2582 scope.go:117] "RemoveContainer" containerID="b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"
Sep 18 19:43:54.117889 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:43:54.117868    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08\": container with ID starting with b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08 not found: ID does not exist" containerID="b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"
Sep 18 19:43:54.117955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.117895    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08"} err="failed to get container status \"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08\": rpc error: code = NotFound desc = could not find container \"b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08\": container with ID starting with b90985bcca6dcfcd4f3a17a01d2e9e1aaec5c5b838f3a73acc0adbd9847e1f08 not found: ID does not exist"
Sep 18 19:43:54.126781 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.126758    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:54.128618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:54.128594    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-8555b79d7d-2m6pm"]
Sep 18 19:43:55.894648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:43:55.894614    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" path="/var/lib/kubelet/pods/18ca41ff-2ff9-4359-8c04-47470493924f/volumes"
Sep 18 19:44:02.650774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:02.650721    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:44:02.659085 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:02.659057    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:44:08.146901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:08.146824    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:44:08.152889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:08.152864    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:44:14.514197 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:14.514109    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:23.171470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:23.171429    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" containerID="cri-o://078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673" gracePeriod=45
Sep 18 19:44:23.173401 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:23.173366    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main" containerID="cri-o://5318d64137f3c2c24ec402a6078a1b705362fd232ca2abd56c2b80aecf86c32e" gracePeriod=45
Sep 18 19:44:23.651813 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:23.651771    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 18 19:44:23.928856 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:23.928833    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:44:24.042213 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042356 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w94dd\" (UniqueName: \"kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042356 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042430 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042482 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042482 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042465    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042583 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042583 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042526    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert\") pod \"805add19-4623-4b7d-be58-c654ff2030c8\" (UID: \"805add19-4623-4b7d-be58-c654ff2030c8\") "
Sep 18 19:44:24.042811 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.042774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:24.043757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.043498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:24.043757 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.043580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home" (OuterVolumeSpecName: "home") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:24.045029 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.044998    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:24.045029 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.045004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd" (OuterVolumeSpecName: "kube-api-access-w94dd") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "kube-api-access-w94dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:24.045175 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.045029    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm" (OuterVolumeSpecName: "dshm") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:24.061379 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.061347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:24.072021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.071994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "805add19-4623-4b7d-be58-c654ff2030c8" (UID: "805add19-4623-4b7d-be58-c654ff2030c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:24.143628 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143605    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143628 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143626    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94dd\" (UniqueName: \"kubernetes.io/projected/805add19-4623-4b7d-be58-c654ff2030c8-kube-api-access-w94dd\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143636    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805add19-4623-4b7d-be58-c654ff2030c8-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143645    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143652    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143660    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143666    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805add19-4623-4b7d-be58-c654ff2030c8-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.143791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.143673    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/805add19-4623-4b7d-be58-c654ff2030c8-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:24.249049 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.249017    2582 generic.go:356] "Generic (PLEG): container finished" podID="805add19-4623-4b7d-be58-c654ff2030c8" containerID="078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673" exitCode=0
Sep 18 19:44:24.249485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.249068    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"
Sep 18 19:44:24.249485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.249078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerDied","Data":"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"}
Sep 18 19:44:24.249485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.249138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6" event={"ID":"805add19-4623-4b7d-be58-c654ff2030c8","Type":"ContainerDied","Data":"66eea907e44b0487d18ceded201483412742c1250fe47205b13be5e52be8b9a8"}
Sep 18 19:44:24.249485 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.249169    2582 scope.go:117] "RemoveContainer" containerID="078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"
Sep 18 19:44:24.265798 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.265777    2582 scope.go:117] "RemoveContainer" containerID="85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"
Sep 18 19:44:24.273685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.273666    2582 scope.go:117] "RemoveContainer" containerID="078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"
Sep 18 19:44:24.273943 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:44:24.273921    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673\": container with ID starting with 078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673 not found: ID does not exist" containerID="078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"
Sep 18 19:44:24.274004 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.273951    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673"} err="failed to get container status \"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673\": rpc error: code = NotFound desc = could not find container \"078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673\": container with ID starting with 078e71d03e5483339125dae6ebbfdb916b09f4adb8a899ff197bd28295c77673 not found: ID does not exist"
Sep 18 19:44:24.274004 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.273968    2582 scope.go:117] "RemoveContainer" containerID="85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"
Sep 18 19:44:24.274236 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:44:24.274216    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8\": container with ID starting with 85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8 not found: ID does not exist" containerID="85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"
Sep 18 19:44:24.274327 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.274242    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8"} err="failed to get container status \"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8\": rpc error: code = NotFound desc = could not find container \"85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8\": container with ID starting with 85c4993483bde381a3a6c65306a856ec721f981b796cfd76b7c9b970ca78bcf8 not found: ID does not exist"
Sep 18 19:44:24.287959 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.287937    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:44:24.296174 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.296153    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7544c47947-65rs6"]
Sep 18 19:44:24.513102 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:24.513024    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:25.894429 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:25.894393    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805add19-4623-4b7d-be58-c654ff2030c8" path="/var/lib/kubelet/pods/805add19-4623-4b7d-be58-c654ff2030c8/volumes"
Sep 18 19:44:29.898342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:29.898316    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:44:29.903126 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:29.903104    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:44:29.910411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:29.905619    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:44:29.915052 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:29.915036    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:44:34.513930 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:34.513892    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:34.514411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:34.514008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:44:44.513163 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:44.513127    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:44:45.330253 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.330213    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:44:45.336567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336538    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main"
Sep 18 19:44:45.336567 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336563    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="storage-initializer"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="storage-initializer"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336596    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="storage-initializer"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336601    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="storage-initializer"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336676    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="main"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336685    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="main"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336767    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ca41ff-2ff9-4359-8c04-47470493924f" containerName="main"
Sep 18 19:44:45.336800 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.336775    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="805add19-4623-4b7d-be58-c654ff2030c8" containerName="main"
Sep 18 19:44:45.341467 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.341448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.349941 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.349914    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 19:44:45.350086 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.349922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 19:44:45.350431 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.350411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:44:45.351005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.350983    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 19:44:45.351483 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.351467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-sxvdz\""
Sep 18 19:44:45.351597 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.351558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:44:45.352475 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.352459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 19:44:45.498318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msf5\" (UniqueName: \"kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.498540 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.498533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599052 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.598977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599052 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msf5\" (UniqueName: \"kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599052 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.599559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.599525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.600123 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.600103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.601762 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.601744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.602023 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.602004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.618945 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.618914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msf5\" (UniqueName: \"kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5\") pod \"tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.653280 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.653252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:45.672618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.672593    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:44:45.676619 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.676600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.707578 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.707548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:44:45.801416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.801386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.801560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.801426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.801560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.801445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.801560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.801472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.902889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.902806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.902889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.902842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.902889 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.902866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.903139 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.902937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.903293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.903268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.903354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.903278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.903577 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.903555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.905539 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.905522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:45.944456 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.944433    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:44:45.946420 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:44:45.946397    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a46ae6_5f8c_4a56_82ce_7cc0e800dd43.slice/crio-85cad89d6b1b9ea7e63d9e562ff95d911b3a63056af9682f6b085134067d9d25 WatchSource:0}: Error finding container 85cad89d6b1b9ea7e63d9e562ff95d911b3a63056af9682f6b085134067d9d25: Status 404 returned error can't find the container with id 85cad89d6b1b9ea7e63d9e562ff95d911b3a63056af9682f6b085134067d9d25
Sep 18 19:44:45.987360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:45.987337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:46.280038 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:46.280010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:44:46.281782 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:44:46.281749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc218a171_da3f_41e4_af60_4ac6c1237158.slice/crio-ff14934a19907728e266674187493727b20dba7e9de5cc49b6fb658f00f281ac WatchSource:0}: Error finding container ff14934a19907728e266674187493727b20dba7e9de5cc49b6fb658f00f281ac: Status 404 returned error can't find the container with id ff14934a19907728e266674187493727b20dba7e9de5cc49b6fb658f00f281ac
Sep 18 19:44:46.358486 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:46.358433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerStarted","Data":"ff14934a19907728e266674187493727b20dba7e9de5cc49b6fb658f00f281ac"}
Sep 18 19:44:46.360464 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:46.360440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerStarted","Data":"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"}
Sep 18 19:44:46.360569 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:46.360495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerStarted","Data":"85cad89d6b1b9ea7e63d9e562ff95d911b3a63056af9682f6b085134067d9d25"}
Sep 18 19:44:47.368895 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:47.368861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerStarted","Data":"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"}
Sep 18 19:44:49.386674 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:49.386636    2582 generic.go:356] "Generic (PLEG): container finished" podID="c218a171-da3f-41e4-af60-4ac6c1237158" containerID="f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b" exitCode=0
Sep 18 19:44:49.387147 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:49.386709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerDied","Data":"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"}
Sep 18 19:44:49.388273 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:49.388254    2582 generic.go:356] "Generic (PLEG): container finished" podID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerID="7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8" exitCode=0
Sep 18 19:44:49.388352 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:49.388282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerDied","Data":"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"}
Sep 18 19:44:50.395306 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.395268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerStarted","Data":"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"}
Sep 18 19:44:50.395767 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.395512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:44:50.397360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.397338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerStarted","Data":"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"}
Sep 18 19:44:50.420545 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.420504    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" podStartSLOduration=2.965241614 podStartE2EDuration="5.420490805s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.455249191s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:50.416913543 +0000 UTC m=+1822.654040553" watchObservedRunningTime="2026-09-18 19:44:50.420490805 +0000 UTC m=+1822.657617838"
Sep 18 19:44:50.443647 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.443606    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" podStartSLOduration=2.997512867 podStartE2EDuration="5.4435948s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.446081933s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:50.441328516 +0000 UTC m=+1822.678455502" watchObservedRunningTime="2026-09-18 19:44:50.4435948 +0000 UTC m=+1822.680721785"
Sep 18 19:44:50.653941 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:50.653862    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:53.420077 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.420043    2582 generic.go:356] "Generic (PLEG): container finished" podID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerID="5318d64137f3c2c24ec402a6078a1b705362fd232ca2abd56c2b80aecf86c32e" exitCode=0
Sep 18 19:44:53.420416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.420106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" event={"ID":"ea3adf1b-bbee-4266-bef5-3ac5fa77611e","Type":"ContainerDied","Data":"5318d64137f3c2c24ec402a6078a1b705362fd232ca2abd56c2b80aecf86c32e"}
Sep 18 19:44:53.420416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.420190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx" event={"ID":"ea3adf1b-bbee-4266-bef5-3ac5fa77611e","Type":"ContainerDied","Data":"b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6"}
Sep 18 19:44:53.420416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.420213    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b881140d4962c4720f536ad5af689ecf3d1075bcc41314e5261770453f0620c6"
Sep 18 19:44:53.422006 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.421977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:44:53.467087 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.467056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs\") pod \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") "
Sep 18 19:44:53.467248 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.467148    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcf9q\" (UniqueName: \"kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q\") pod \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\" (UID: \"ea3adf1b-bbee-4266-bef5-3ac5fa77611e\") "
Sep 18 19:44:53.469342 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.469309    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea3adf1b-bbee-4266-bef5-3ac5fa77611e" (UID: "ea3adf1b-bbee-4266-bef5-3ac5fa77611e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:53.469635 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.469614    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q" (OuterVolumeSpecName: "kube-api-access-kcf9q") pod "ea3adf1b-bbee-4266-bef5-3ac5fa77611e" (UID: "ea3adf1b-bbee-4266-bef5-3ac5fa77611e"). InnerVolumeSpecName "kube-api-access-kcf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:53.568236 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.568192    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:53.568236 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:53.568225    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcf9q\" (UniqueName: \"kubernetes.io/projected/ea3adf1b-bbee-4266-bef5-3ac5fa77611e-kube-api-access-kcf9q\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:44:54.425397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:54.425365    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"
Sep 18 19:44:54.445996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:54.445967    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:44:54.451281 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:54.451258    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-75sjx"]
Sep 18 19:44:55.653570 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:55.653538    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:55.666278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:55.666250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:44:55.895315 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:55.895275    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" path="/var/lib/kubelet/pods/ea3adf1b-bbee-4266-bef5-3ac5fa77611e/volumes"
Sep 18 19:44:56.449097 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:44:56.449070    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:45:21.415505 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:21.415466    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:45:43.044714 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:43.044638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp_c218a171-da3f-41e4-af60-4ac6c1237158/main/0.log"
Sep 18 19:45:44.576455 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.576418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:45:44.576853 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.576679    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="main" containerID="cri-o://587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329" gracePeriod=30
Sep 18 19:45:44.582921 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.582889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:45:44.591886 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:44.591843    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:45:44.591886 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.591886 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.591886 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd" cmd=["/bin/sleep","15"]
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.591914    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" message="" err=<
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:44.591930    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="main" containerID="cri-o://b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"
Sep 18 19:45:44.592097 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.591971    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="main" containerID="cri-o://b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd" gracePeriod=60
Sep 18 19:45:44.933946 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.933918    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:45:44.981209 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6msf5\" (UniqueName: \"kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981395 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache\") pod \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\" (UID: \"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43\") "
Sep 18 19:45:44.981650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home" (OuterVolumeSpecName: "home") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:44.981650 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:44.981883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981712    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:44.981883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981756    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:44.981883 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.981820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache" (OuterVolumeSpecName: "model-cache") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:44.982224 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.982016    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:44.983794 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.983760    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm" (OuterVolumeSpecName: "dshm") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:44.984129 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.984104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:44.984231 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:44.984214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5" (OuterVolumeSpecName: "kube-api-access-6msf5") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "kube-api-access-6msf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:45.010669 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.010635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" (UID: "97a46ae6-5f8c-4a56-82ce-7cc0e800dd43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:45.083237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083203    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.083237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083235    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.083237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083247    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6msf5\" (UniqueName: \"kubernetes.io/projected/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kube-api-access-6msf5\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.083237 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083256    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.083520 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083265    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.083520 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.083273    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.208129 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.208099    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:45:45.284451 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284422    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir\") pod \"c218a171-da3f-41e4-af60-4ac6c1237158\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") "
Sep 18 19:45:45.284607 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284467    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs\") pod \"c218a171-da3f-41e4-af60-4ac6c1237158\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") "
Sep 18 19:45:45.284607 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284486    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert\") pod \"c218a171-da3f-41e4-af60-4ac6c1237158\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") "
Sep 18 19:45:45.284607 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284523    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location\") pod \"c218a171-da3f-41e4-af60-4ac6c1237158\" (UID: \"c218a171-da3f-41e4-af60-4ac6c1237158\") "
Sep 18 19:45:45.284914 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c218a171-da3f-41e4-af60-4ac6c1237158" (UID: "c218a171-da3f-41e4-af60-4ac6c1237158"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:45.284914 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.284905    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c218a171-da3f-41e4-af60-4ac6c1237158" (UID: "c218a171-da3f-41e4-af60-4ac6c1237158"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:45.286670 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.286635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c218a171-da3f-41e4-af60-4ac6c1237158" (UID: "c218a171-da3f-41e4-af60-4ac6c1237158"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:45.308041 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.308004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c218a171-da3f-41e4-af60-4ac6c1237158" (UID: "c218a171-da3f-41e4-af60-4ac6c1237158"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:45.386021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.385991    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.386021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.386017    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c218a171-da3f-41e4-af60-4ac6c1237158-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.386021 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.386027    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c218a171-da3f-41e4-af60-4ac6c1237158-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.386263 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.386035    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c218a171-da3f-41e4-af60-4ac6c1237158-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:45:45.686349 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.686315    2582 generic.go:356] "Generic (PLEG): container finished" podID="c218a171-da3f-41e4-af60-4ac6c1237158" containerID="587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329" exitCode=0
Sep 18 19:45:45.686821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.686372    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"
Sep 18 19:45:45.686821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.686396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerDied","Data":"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"}
Sep 18 19:45:45.686821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.686450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp" event={"ID":"c218a171-da3f-41e4-af60-4ac6c1237158","Type":"ContainerDied","Data":"ff14934a19907728e266674187493727b20dba7e9de5cc49b6fb658f00f281ac"}
Sep 18 19:45:45.686821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.686479    2582 scope.go:117] "RemoveContainer" containerID="587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"
Sep 18 19:45:45.688186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.688160    2582 generic.go:356] "Generic (PLEG): container finished" podID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerID="b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd" exitCode=0
Sep 18 19:45:45.688296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.688212    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"
Sep 18 19:45:45.688296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.688221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerDied","Data":"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"}
Sep 18 19:45:45.688296 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.688258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m" event={"ID":"97a46ae6-5f8c-4a56-82ce-7cc0e800dd43","Type":"ContainerDied","Data":"85cad89d6b1b9ea7e63d9e562ff95d911b3a63056af9682f6b085134067d9d25"}
Sep 18 19:45:45.701191 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.701169    2582 scope.go:117] "RemoveContainer" containerID="f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"
Sep 18 19:45:45.712489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.712472    2582 scope.go:117] "RemoveContainer" containerID="587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"
Sep 18 19:45:45.712760 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:45.712720    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329\": container with ID starting with 587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329 not found: ID does not exist" containerID="587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"
Sep 18 19:45:45.712850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.712764    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329"} err="failed to get container status \"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329\": rpc error: code = NotFound desc = could not find container \"587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329\": container with ID starting with 587a08a9a10a2abab09e6e8a820159e7c73c79ad297cc2f8cc15538d05158329 not found: ID does not exist"
Sep 18 19:45:45.712850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.712778    2582 scope.go:117] "RemoveContainer" containerID="f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"
Sep 18 19:45:45.713009 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:45.712990    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b\": container with ID starting with f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b not found: ID does not exist" containerID="f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"
Sep 18 19:45:45.713054 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.713015    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b"} err="failed to get container status \"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b\": rpc error: code = NotFound desc = could not find container \"f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b\": container with ID starting with f4abc4694c7776b9cdaafc082618f32e988b34824b10c573aeb8aefd8852937b not found: ID does not exist"
Sep 18 19:45:45.713054 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.713030    2582 scope.go:117] "RemoveContainer" containerID="b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"
Sep 18 19:45:45.717779 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.717756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:45:45.722410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.722393    2582 scope.go:117] "RemoveContainer" containerID="7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"
Sep 18 19:45:45.723025 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.723004    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5bdc4cd48d-trfcp"]
Sep 18 19:45:45.732105 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.731980    2582 scope.go:117] "RemoveContainer" containerID="b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"
Sep 18 19:45:45.732429 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:45.732396    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd\": container with ID starting with b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd not found: ID does not exist" containerID="b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"
Sep 18 19:45:45.732497 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.732440    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd"} err="failed to get container status \"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd\": rpc error: code = NotFound desc = could not find container \"b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd\": container with ID starting with b2922ceb63c09193e6523f36306bc28aa18d08ed494f9d101defabbd467c7cbd not found: ID does not exist"
Sep 18 19:45:45.732497 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.732457    2582 scope.go:117] "RemoveContainer" containerID="7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"
Sep 18 19:45:45.732783 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:45:45.732765    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8\": container with ID starting with 7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8 not found: ID does not exist" containerID="7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"
Sep 18 19:45:45.732877 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.732787    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8"} err="failed to get container status \"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8\": rpc error: code = NotFound desc = could not find container \"7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8\": container with ID starting with 7956a6df0019fa7d41b5494652620e3ce86b0778f51a760437193bfa8d3db4c8 not found: ID does not exist"
Sep 18 19:45:45.737251 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.737228    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:45:45.740019 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.740000    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-c6f9494cd-wbj7m"]
Sep 18 19:45:45.894986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.894945    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" path="/var/lib/kubelet/pods/97a46ae6-5f8c-4a56-82ce-7cc0e800dd43/volumes"
Sep 18 19:45:45.895382 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:45:45.895369    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" path="/var/lib/kubelet/pods/c218a171-da3f-41e4-af60-4ac6c1237158/volumes"
Sep 18 19:46:44.726177 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726145    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726424    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="storage-initializer"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="storage-initializer"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726452    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="storage-initializer"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726457    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="storage-initializer"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726469    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726476    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726481    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726488    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726493    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726544    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a46ae6-5f8c-4a56-82ce-7cc0e800dd43" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726552    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3adf1b-bbee-4266-bef5-3ac5fa77611e" containerName="main"
Sep 18 19:46:44.726618 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.726559    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c218a171-da3f-41e4-af60-4ac6c1237158" containerName="main"
Sep 18 19:46:44.730798 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.730778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.733578 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.733556    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:46:44.733687 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.733599    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 19:46:44.734530 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.734509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 19:46:44.734827 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.734589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fbm2d\""
Sep 18 19:46:44.734827 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.734802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 19:46:44.734952 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.734841    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 19:46:44.745403 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.745380    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:46:44.812789 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.812908 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.812908 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.812908 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.813017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.813017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.813017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.812988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.813017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.813004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxs5l\" (UniqueName: \"kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913340 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913600 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913600 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxs5l\" (UniqueName: \"kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913600 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913600 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913830 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.913981 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.914054 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.913994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.914114 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.914094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.914384 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.914369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.915917 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.915899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.916165 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.916131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.923245 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.923220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxs5l\" (UniqueName: \"kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l\") pod \"tokenizer-template-args-test-kserve-6475d67bc4-2qrwv\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:44.991846 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.991768    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:46:44.995320 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:44.995300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.011133 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.011110    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:46:45.041243 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.041212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:45.115411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.115383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.115561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.115422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.115561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.115495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.115561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.115552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216345 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216516 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216516 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216516 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.216901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.216881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.217098 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.217082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.219023 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.219003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.306201 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.306132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:45.316973 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.316933    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:46:45.319142 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:46:45.319115    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426307bf_5f59_4864_911e_74581d0e88b6.slice/crio-668922f5554589bceabb62d785f8c6759a1bcc1cf7596257375d23dbf8014a2e WatchSource:0}: Error finding container 668922f5554589bceabb62d785f8c6759a1bcc1cf7596257375d23dbf8014a2e: Status 404 returned error can't find the container with id 668922f5554589bceabb62d785f8c6759a1bcc1cf7596257375d23dbf8014a2e
Sep 18 19:46:45.586470 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.586402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:46:45.588985 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:46:45.588955    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a926f4f_e8a0_44ca_aa5b_468d1e2d0b9a.slice/crio-06f634c165ccff3e9d8311d34763b133d5ae5fd4536ff66695d0b7daa48c99a5 WatchSource:0}: Error finding container 06f634c165ccff3e9d8311d34763b133d5ae5fd4536ff66695d0b7daa48c99a5: Status 404 returned error can't find the container with id 06f634c165ccff3e9d8311d34763b133d5ae5fd4536ff66695d0b7daa48c99a5
Sep 18 19:46:45.975682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.975640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerStarted","Data":"bc3b5952c534207e2743b9624de746f5464fb3556d44a4dd0d3de9739a8fcee2"}
Sep 18 19:46:45.976208 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.976159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerStarted","Data":"668922f5554589bceabb62d785f8c6759a1bcc1cf7596257375d23dbf8014a2e"}
Sep 18 19:46:45.977684 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.977652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerStarted","Data":"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"}
Sep 18 19:46:45.977817 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:45.977706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerStarted","Data":"06f634c165ccff3e9d8311d34763b133d5ae5fd4536ff66695d0b7daa48c99a5"}
Sep 18 19:46:47.989560 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:47.989523    2582 generic.go:356] "Generic (PLEG): container finished" podID="426307bf-5f59-4864-911e-74581d0e88b6" containerID="bc3b5952c534207e2743b9624de746f5464fb3556d44a4dd0d3de9739a8fcee2" exitCode=0
Sep 18 19:46:47.990033 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:47.989668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerDied","Data":"bc3b5952c534207e2743b9624de746f5464fb3556d44a4dd0d3de9739a8fcee2"}
Sep 18 19:46:48.996056 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:48.996023    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerID="1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159" exitCode=0
Sep 18 19:46:48.996505 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:48.996095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerDied","Data":"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"}
Sep 18 19:46:48.998180 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:48.998157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerStarted","Data":"3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79"}
Sep 18 19:46:49.041355 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:49.041298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:49.044186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:49.044134    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" podStartSLOduration=2.587222647 podStartE2EDuration="5.044115925s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.456893278s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:49.041185369 +0000 UTC m=+1941.278312371" watchObservedRunningTime="2026-09-18 19:46:49.044115925 +0000 UTC m=+1941.281242911"
Sep 18 19:46:50.005850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:50.005806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerStarted","Data":"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"}
Sep 18 19:46:50.006289 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:50.006269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:46:50.030014 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:50.029973    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" podStartSLOduration=3.536276471 podStartE2EDuration="6.029958698s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.493682227s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:50.028784172 +0000 UTC m=+1942.265911157" watchObservedRunningTime="2026-09-18 19:46:50.029958698 +0000 UTC m=+1942.267085676"
Sep 18 19:46:55.041533 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:55.041500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:55.054201 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:55.054175    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:46:56.046668 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:46:56.046635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:47:21.023888 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:21.023806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:47:22.454840 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.454803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c_7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a/main/0.log"
Sep 18 19:47:22.855532 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.855493    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:47:22.855798 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.855771    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="main" containerID="cri-o://b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4" gracePeriod=30
Sep 18 19:47:22.858549 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.858521    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:47:22.867627 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:47:22.867587    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:47:22.867627 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.867627 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.867627 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79" cmd=["/bin/sleep","15"]
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.867647    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" message="" err=<
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:47:22.867663    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main" containerID="cri-o://3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79"
Sep 18 19:47:22.867907 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:22.867716    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main" containerID="cri-o://3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79" gracePeriod=60
Sep 18 19:47:23.036245 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.036207    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 19:47:23.167631 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.167594    2582 generic.go:356] "Generic (PLEG): container finished" podID="426307bf-5f59-4864-911e-74581d0e88b6" containerID="3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79" exitCode=0
Sep 18 19:47:23.167826 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.167665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerDied","Data":"3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79"}
Sep 18 19:47:23.206467 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.206437    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:47:23.321536 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321568    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321648    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321665    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321687    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxs5l\" (UniqueName: \"kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l\") pod \"426307bf-5f59-4864-911e-74581d0e88b6\" (UID: \"426307bf-5f59-4864-911e-74581d0e88b6\") "
Sep 18 19:47:23.321957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.322115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.321994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:23.322115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.322096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.322218 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.322182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home" (OuterVolumeSpecName: "home") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.326900 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.326878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm" (OuterVolumeSpecName: "dshm") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.329906 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.329885    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l" (OuterVolumeSpecName: "kube-api-access-mxs5l") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "kube-api-access-mxs5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:23.329906 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.329889    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:23.345083 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.345058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "426307bf-5f59-4864-911e-74581d0e88b6" (UID: "426307bf-5f59-4864-911e-74581d0e88b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.422940 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422836    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.422940 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422871    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426307bf-5f59-4864-911e-74581d0e88b6-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.422940 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422914    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.422940 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422930    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/426307bf-5f59-4864-911e-74581d0e88b6-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.422940 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422943    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.423270 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422955    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.423270 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422975    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426307bf-5f59-4864-911e-74581d0e88b6-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.423270 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.422985    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxs5l\" (UniqueName: \"kubernetes.io/projected/426307bf-5f59-4864-911e-74581d0e88b6-kube-api-access-mxs5l\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.488230 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.488207    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:47:23.523348 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert\") pod \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") "
Sep 18 19:47:23.523475 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir\") pod \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") "
Sep 18 19:47:23.523475 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location\") pod \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") "
Sep 18 19:47:23.523601 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs\") pod \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\" (UID: \"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a\") "
Sep 18 19:47:23.523794 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" (UID: "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:23.523879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.523848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" (UID: "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.525669 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.525647    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" (UID: "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:23.545284 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.545252    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" (UID: "7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:23.624812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.624785    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.624812 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.624809    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.624944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.624818    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.624944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:23.624826    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:47:24.174119 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.174036    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"
Sep 18 19:47:24.174119 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.174064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv" event={"ID":"426307bf-5f59-4864-911e-74581d0e88b6","Type":"ContainerDied","Data":"668922f5554589bceabb62d785f8c6759a1bcc1cf7596257375d23dbf8014a2e"}
Sep 18 19:47:24.174326 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.174127    2582 scope.go:117] "RemoveContainer" containerID="3cbfcda9d495371a4aae26f5be65be3785c025261f890de6a0812772658d2b79"
Sep 18 19:47:24.175962 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.175935    2582 generic.go:356] "Generic (PLEG): container finished" podID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerID="b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4" exitCode=0
Sep 18 19:47:24.176081 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.175973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerDied","Data":"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"}
Sep 18 19:47:24.176081 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.176017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c" event={"ID":"7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a","Type":"ContainerDied","Data":"06f634c165ccff3e9d8311d34763b133d5ae5fd4536ff66695d0b7daa48c99a5"}
Sep 18 19:47:24.176499 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.176290    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"
Sep 18 19:47:24.183551 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.183535    2582 scope.go:117] "RemoveContainer" containerID="bc3b5952c534207e2743b9624de746f5464fb3556d44a4dd0d3de9739a8fcee2"
Sep 18 19:47:24.191416 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.191358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:47:24.191463 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.191442    2582 scope.go:117] "RemoveContainer" containerID="b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"
Sep 18 19:47:24.197266 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.197245    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7bb4df7f64-l7v7c"]
Sep 18 19:47:24.198998 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.198984    2582 scope.go:117] "RemoveContainer" containerID="1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"
Sep 18 19:47:24.206388 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.206357    2582 scope.go:117] "RemoveContainer" containerID="b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"
Sep 18 19:47:24.206666 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:47:24.206635    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4\": container with ID starting with b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4 not found: ID does not exist" containerID="b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"
Sep 18 19:47:24.206774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.206675    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4"} err="failed to get container status \"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4\": rpc error: code = NotFound desc = could not find container \"b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4\": container with ID starting with b4a15abb3590ef006207e10811b8040f41fa0e2a2f8752ed13178a99437192d4 not found: ID does not exist"
Sep 18 19:47:24.206774 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.206697    2582 scope.go:117] "RemoveContainer" containerID="1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"
Sep 18 19:47:24.207005 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:47:24.206987    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159\": container with ID starting with 1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159 not found: ID does not exist" containerID="1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"
Sep 18 19:47:24.207066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.207010    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159"} err="failed to get container status \"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159\": rpc error: code = NotFound desc = could not find container \"1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159\": container with ID starting with 1e464b768ed22f92b60e22d9d4277455a43154582fda375ea363f3703be66159 not found: ID does not exist"
Sep 18 19:47:24.208504 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.208487    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:47:24.213000 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:24.212977    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6475d67bc4-2qrwv"]
Sep 18 19:47:25.894624 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:25.894595    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426307bf-5f59-4864-911e-74581d0e88b6" path="/var/lib/kubelet/pods/426307bf-5f59-4864-911e-74581d0e88b6/volumes"
Sep 18 19:47:25.895040 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:47:25.895024    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" path="/var/lib/kubelet/pods/7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a/volumes"
Sep 18 19:48:26.133586 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133545    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133874    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="storage-initializer"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133886    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="storage-initializer"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133902    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133908    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133917    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="main"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133922    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="main"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133928    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="storage-initializer"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133933    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="storage-initializer"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133980    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="426307bf-5f59-4864-911e-74581d0e88b6" containerName="main"
Sep 18 19:48:26.134047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.133987    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a926f4f-e8a0-44ca-aa5b-468d1e2d0b9a" containerName="main"
Sep 18 19:48:26.136805 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.136787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.139893 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.139871    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 19:48:26.140785 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.140762    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 19:48:26.140923 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.140877    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-mgpxv\""
Sep 18 19:48:26.140923 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.140911    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 19:48:26.149944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.149921    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:48:26.267652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.262411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.267888 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.267861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2wpt\" (UniqueName: \"kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.267951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.267922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.268006 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.267945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.268058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.268008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.268196 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.268173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369338 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2wpt\" (UniqueName: \"kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.369754 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.369717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.370188 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.370165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.370319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.370253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.370474 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.370451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.372106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.372082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.372278 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.372255    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:48:26.372743 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.372711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.376072 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.376056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.379690 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.379654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-2s6fg\""
Sep 18 19:48:26.383963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.383915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2wpt\" (UniqueName: \"kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.396919 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.396900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:48:26.446857 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.446831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:26.470434 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.470411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.470562 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.470466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg9w\" (UniqueName: \"kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.570935 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.570898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg9w\" (UniqueName: \"kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.571108 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.571012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.573590 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.573566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.579926 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.579908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg9w\" (UniqueName: \"kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.700457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.700396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:26.723364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.723333    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:48:26.725186 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:48:26.725163    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b25952d_e1ef_44f0_b80c_f6c9608460c0.slice/crio-21253ec5a7107101edb0957655ba754ee6ad8bca05a28c901525b824b5681b2d WatchSource:0}: Error finding container 21253ec5a7107101edb0957655ba754ee6ad8bca05a28c901525b824b5681b2d: Status 404 returned error can't find the container with id 21253ec5a7107101edb0957655ba754ee6ad8bca05a28c901525b824b5681b2d
Sep 18 19:48:26.987356 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:26.987116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:48:26.990481 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:48:26.990447    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf129f0eb_7cb2_4411_a2dd_b239e816b170.slice/crio-6df24ea84759508d2dd74023945dd2506740e4c179e27772dbad9fecc7d650cc WatchSource:0}: Error finding container 6df24ea84759508d2dd74023945dd2506740e4c179e27772dbad9fecc7d650cc: Status 404 returned error can't find the container with id 6df24ea84759508d2dd74023945dd2506740e4c179e27772dbad9fecc7d650cc
Sep 18 19:48:27.470787 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.470741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" event={"ID":"f129f0eb-7cb2-4411-a2dd-b239e816b170","Type":"ContainerStarted","Data":"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"}
Sep 18 19:48:27.471423 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.470814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:48:27.471423 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.470838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" event={"ID":"f129f0eb-7cb2-4411-a2dd-b239e816b170","Type":"ContainerStarted","Data":"6df24ea84759508d2dd74023945dd2506740e4c179e27772dbad9fecc7d650cc"}
Sep 18 19:48:27.472422 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.472396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" event={"ID":"3b25952d-e1ef-44f0-b80c-f6c9608460c0","Type":"ContainerStarted","Data":"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"}
Sep 18 19:48:27.472555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.472438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" event={"ID":"3b25952d-e1ef-44f0-b80c-f6c9608460c0","Type":"ContainerStarted","Data":"21253ec5a7107101edb0957655ba754ee6ad8bca05a28c901525b824b5681b2d"}
Sep 18 19:48:27.533607 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.533546    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podStartSLOduration=1.533529305 podStartE2EDuration="1.533529305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:27.530607207 +0000 UTC m=+2039.767734206" watchObservedRunningTime="2026-09-18 19:48:27.533529305 +0000 UTC m=+2039.770656290"
Sep 18 19:48:27.565839 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:27.565790    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" podStartSLOduration=1.565775038 podStartE2EDuration="1.565775038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:27.565013573 +0000 UTC m=+2039.802140562" watchObservedRunningTime="2026-09-18 19:48:27.565775038 +0000 UTC m=+2039.802902023"
Sep 18 19:48:28.447633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:28.447596    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:30.044749 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:30.044703    2582 scope.go:117] "RemoveContainer" containerID="5318d64137f3c2c24ec402a6078a1b705362fd232ca2abd56c2b80aecf86c32e"
Sep 18 19:48:36.447662 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:36.447623    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:36.461631 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:36.461602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:36.525932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:36.525909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:48:58.479050 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:48:58.478974    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:49:00.583038 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:00.583001    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:49:00.597185 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:00.597155    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:49:00.606414 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:49:00.606370    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:49:00.606414 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:49:00.606414 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:49:00.606414 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f" cmd=["/bin/sleep","15"]
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:00.606435    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" message="" err=<
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:49:00.606452    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerName="main" containerID="cri-o://e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"
Sep 18 19:49:00.606719 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:00.606499    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerName="main" containerID="cri-o://e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f" gracePeriod=60
Sep 18 19:49:01.346325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.346300    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:49:01.439701 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.439901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2wpt\" (UniqueName: \"kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.439901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439806    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.439901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.439901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.439901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm\") pod \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\" (UID: \"3b25952d-e1ef-44f0-b80c-f6c9608460c0\") "
Sep 18 19:49:01.440182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.439947    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home" (OuterVolumeSpecName: "home") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:01.440182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.440118    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:01.440182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.440143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:01.440339 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.440192    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.440339 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.440209    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.442256 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.442225    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt" (OuterVolumeSpecName: "kube-api-access-m2wpt") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "kube-api-access-m2wpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:49:01.442377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.442253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:49:01.442377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.442259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm" (OuterVolumeSpecName: "dshm") pod "3b25952d-e1ef-44f0-b80c-f6c9608460c0" (UID: "3b25952d-e1ef-44f0-b80c-f6c9608460c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:01.541561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.541486    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.541561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.541513    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.541561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.541522    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2wpt\" (UniqueName: \"kubernetes.io/projected/3b25952d-e1ef-44f0-b80c-f6c9608460c0-kube-api-access-m2wpt\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.541561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.541530    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b25952d-e1ef-44f0-b80c-f6c9608460c0-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:01.640483 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.640440    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerID="e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f" exitCode=0
Sep 18 19:49:01.640983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.640533    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"
Sep 18 19:49:01.640983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.640564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" event={"ID":"3b25952d-e1ef-44f0-b80c-f6c9608460c0","Type":"ContainerDied","Data":"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"}
Sep 18 19:49:01.640983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.640626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb" event={"ID":"3b25952d-e1ef-44f0-b80c-f6c9608460c0","Type":"ContainerDied","Data":"21253ec5a7107101edb0957655ba754ee6ad8bca05a28c901525b824b5681b2d"}
Sep 18 19:49:01.640983 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.640653    2582 scope.go:117] "RemoveContainer" containerID="e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"
Sep 18 19:49:01.650016 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.650000    2582 scope.go:117] "RemoveContainer" containerID="e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"
Sep 18 19:49:01.650253 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:49:01.650237    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f\": container with ID starting with e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f not found: ID does not exist" containerID="e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"
Sep 18 19:49:01.650292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.650261    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f"} err="failed to get container status \"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f\": rpc error: code = NotFound desc = could not find container \"e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f\": container with ID starting with e9be83d9d749f038d061a121a8f0b8214f4eea8528014e9ff276af8f3c27ec2f not found: ID does not exist"
Sep 18 19:49:01.665527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.665502    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:49:01.669611 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.669592    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ssqgb"]
Sep 18 19:49:01.895434 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:01.895356    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" path="/var/lib/kubelet/pods/3b25952d-e1ef-44f0-b80c-f6c9608460c0/volumes"
Sep 18 19:49:08.479303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:08.479264    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:49:15.604951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:15.604902    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main" containerID="cri-o://afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77" gracePeriod=45
Sep 18 19:49:18.478888 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:18.478848    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:49:28.479756 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:28.479696    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:49:28.480143 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:28.479820    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:49:29.934875 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:29.934848    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:49:29.940504 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:29.940483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:49:29.942122 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:29.942100    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:49:29.947346 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:29.947328    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:49:38.479694 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:38.479650    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:49:45.841438 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.841405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:49:45.847291 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.847268    2582 generic.go:356] "Generic (PLEG): container finished" podID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerID="afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77" exitCode=0
Sep 18 19:49:45.847405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.847304    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"
Sep 18 19:49:45.847405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.847348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" event={"ID":"f129f0eb-7cb2-4411-a2dd-b239e816b170","Type":"ContainerDied","Data":"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"}
Sep 18 19:49:45.847405 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.847397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj" event={"ID":"f129f0eb-7cb2-4411-a2dd-b239e816b170","Type":"ContainerDied","Data":"6df24ea84759508d2dd74023945dd2506740e4c179e27772dbad9fecc7d650cc"}
Sep 18 19:49:45.847522 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.847421    2582 scope.go:117] "RemoveContainer" containerID="afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"
Sep 18 19:49:45.855720 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.855704    2582 scope.go:117] "RemoveContainer" containerID="afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"
Sep 18 19:49:45.856036 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:49:45.856015    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77\": container with ID starting with afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77 not found: ID does not exist" containerID="afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"
Sep 18 19:49:45.856093 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.856046    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77"} err="failed to get container status \"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77\": rpc error: code = NotFound desc = could not find container \"afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77\": container with ID starting with afe774089838bc266fec0fbe498b00d65d8244ad282908cffe7de885abc0ac77 not found: ID does not exist"
Sep 18 19:49:45.958555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.958488    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs\") pod \"f129f0eb-7cb2-4411-a2dd-b239e816b170\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") "
Sep 18 19:49:45.958679 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.958559    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gg9w\" (UniqueName: \"kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w\") pod \"f129f0eb-7cb2-4411-a2dd-b239e816b170\" (UID: \"f129f0eb-7cb2-4411-a2dd-b239e816b170\") "
Sep 18 19:49:45.960770 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.960744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f129f0eb-7cb2-4411-a2dd-b239e816b170" (UID: "f129f0eb-7cb2-4411-a2dd-b239e816b170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:49:45.960880 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:45.960784    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w" (OuterVolumeSpecName: "kube-api-access-4gg9w") pod "f129f0eb-7cb2-4411-a2dd-b239e816b170" (UID: "f129f0eb-7cb2-4411-a2dd-b239e816b170"). InnerVolumeSpecName "kube-api-access-4gg9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:49:46.059835 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:46.059808    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gg9w\" (UniqueName: \"kubernetes.io/projected/f129f0eb-7cb2-4411-a2dd-b239e816b170-kube-api-access-4gg9w\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:46.059835 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:46.059828    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f129f0eb-7cb2-4411-a2dd-b239e816b170-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:49:46.171698 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:46.171667    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:49:46.176532 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:46.176510    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezzdvj"]
Sep 18 19:49:47.893969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:47.893938    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" path="/var/lib/kubelet/pods/f129f0eb-7cb2-4411-a2dd-b239e816b170/volumes"
Sep 18 19:49:57.605287 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605249    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:49:57.605678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605643    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerName="main"
Sep 18 19:49:57.605678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605661    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerName="main"
Sep 18 19:49:57.605678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605674    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main"
Sep 18 19:49:57.605832 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main"
Sep 18 19:49:57.605832 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605785    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b25952d-e1ef-44f0-b80c-f6c9608460c0" containerName="main"
Sep 18 19:49:57.605832 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.605793    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f129f0eb-7cb2-4411-a2dd-b239e816b170" containerName="main"
Sep 18 19:49:57.608566 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.608547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.611978 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.611956    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 19:49:57.611978 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.611971    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 19:49:57.612124 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.612024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-d9fgh\""
Sep 18 19:49:57.612124 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.611981    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 19:49:57.623765 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.623719    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:49:57.742510 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.742510 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.742751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.742751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.742751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.742751 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.742697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97q7\" (UniqueName: \"kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.843996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.843967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.843996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.843998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97q7\" (UniqueName: \"kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844473 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844548 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.844548 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.844500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.846326 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.846306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.846627 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.846610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.856186 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.856137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97q7\" (UniqueName: \"kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:57.897421 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.897387    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:49:57.900684 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.900664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:57.903047 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.903025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rm64h\""
Sep 18 19:49:57.915063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.915043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:49:57.925064 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:57.925046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:58.045695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.045662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.045860 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.045757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6sz\" (UniqueName: \"kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.146898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.146826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.146898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.146894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6sz\" (UniqueName: \"kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.149522 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.149498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.155879 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.155851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6sz\" (UniqueName: \"kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.203066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.202954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:49:58.205088 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:49:58.205068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75185dce_79c6_4b3e_bab9_d239d6b635ed.slice/crio-e4dee61eac452f90a6d7b7a8a7d9efbfe85f78f30bb43ea090abc7f3f46a61ba WatchSource:0}: Error finding container e4dee61eac452f90a6d7b7a8a7d9efbfe85f78f30bb43ea090abc7f3f46a61ba: Status 404 returned error can't find the container with id e4dee61eac452f90a6d7b7a8a7d9efbfe85f78f30bb43ea090abc7f3f46a61ba
Sep 18 19:49:58.210830 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.210810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.518655 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.518623    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:49:58.522898 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:49:58.522861    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523296e3_9dc8_4f5a_a719_5f72d39f9e55.slice/crio-c31af23e0ba3d341e7d7816f32af9c782583d6cdd00b09807dff1881fafa4c50 WatchSource:0}: Error finding container c31af23e0ba3d341e7d7816f32af9c782583d6cdd00b09807dff1881fafa4c50: Status 404 returned error can't find the container with id c31af23e0ba3d341e7d7816f32af9c782583d6cdd00b09807dff1881fafa4c50
Sep 18 19:49:58.910756 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.910627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" event={"ID":"75185dce-79c6-4b3e-bab9-d239d6b635ed","Type":"ContainerStarted","Data":"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"}
Sep 18 19:49:58.910756 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.910693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" event={"ID":"75185dce-79c6-4b3e-bab9-d239d6b635ed","Type":"ContainerStarted","Data":"e4dee61eac452f90a6d7b7a8a7d9efbfe85f78f30bb43ea090abc7f3f46a61ba"}
Sep 18 19:49:58.912260 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.912231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" event={"ID":"523296e3-9dc8-4f5a-a719-5f72d39f9e55","Type":"ContainerStarted","Data":"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"}
Sep 18 19:49:58.912372 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.912283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:49:58.912372 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.912302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" event={"ID":"523296e3-9dc8-4f5a-a719-5f72d39f9e55","Type":"ContainerStarted","Data":"c31af23e0ba3d341e7d7816f32af9c782583d6cdd00b09807dff1881fafa4c50"}
Sep 18 19:49:58.925163 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:58.925140    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:49:59.120678 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:59.120623    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" podStartSLOduration=2.120606442 podStartE2EDuration="2.120606442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:59.049256483 +0000 UTC m=+2131.286383469" watchObservedRunningTime="2026-09-18 19:49:59.120606442 +0000 UTC m=+2131.357733429"
Sep 18 19:49:59.121251 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:49:59.121214    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podStartSLOduration=2.121204619 podStartE2EDuration="2.121204619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:59.113295852 +0000 UTC m=+2131.350422837" watchObservedRunningTime="2026-09-18 19:49:59.121204619 +0000 UTC m=+2131.358331604"
Sep 18 19:50:07.925872 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:07.925830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:50:07.938366 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:07.938341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:50:07.965496 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:07.965468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:50:29.919058 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:29.919029    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:50:43.514125 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.514092    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:50:43.518535 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.518517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.521923 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.521904    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 19:50:43.534692 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.534668    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:50:43.605617 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.605617 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.605876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw856\" (UniqueName: \"kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.605876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.605876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.605876 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.605837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706686 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706686 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw856\" (UniqueName: \"kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.706915 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.706844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.707238 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.707215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.707302 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.707273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.707302 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.707273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.709180 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.709155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.709487 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.709470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.716537 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.716513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw856\" (UniqueName: \"kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.820957 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.820887    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:50:43.824263 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.824245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:43.826843 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.826824    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-jg772\""
Sep 18 19:50:43.828075 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.828060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:43.842453 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.842427    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:50:43.908799 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.908771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxvd\" (UniqueName: \"kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:43.908951 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:43.908915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.010082 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.010053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.010259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.010101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxvd\" (UniqueName: \"kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.012671 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.012647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.020011 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.019989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxvd\" (UniqueName: \"kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.124950 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.124927    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:50:44.127254 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:50:44.127227    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec6554e_1bf9_4e2a_a37c_d4fc41412a15.slice/crio-111058b4451ce9b62156996aa3a67123f7060c906c8a8d104918c3f85dea4eec WatchSource:0}: Error finding container 111058b4451ce9b62156996aa3a67123f7060c906c8a8d104918c3f85dea4eec: Status 404 returned error can't find the container with id 111058b4451ce9b62156996aa3a67123f7060c906c8a8d104918c3f85dea4eec
Sep 18 19:50:44.135005 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.134988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:44.418090 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:44.418065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:50:44.420064 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:50:44.420034    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc486b62e_ecb2_4dd9_a4b1_7d4b5d205920.slice/crio-3c3c445fb094ed8b6ee927498d8c0f2570944e0505b6c3638059df2913b7ab3a WatchSource:0}: Error finding container 3c3c445fb094ed8b6ee927498d8c0f2570944e0505b6c3638059df2913b7ab3a: Status 404 returned error can't find the container with id 3c3c445fb094ed8b6ee927498d8c0f2570944e0505b6c3638059df2913b7ab3a
Sep 18 19:50:45.134110 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.134061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" event={"ID":"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15","Type":"ContainerStarted","Data":"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"}
Sep 18 19:50:45.134586 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.134127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" event={"ID":"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15","Type":"ContainerStarted","Data":"111058b4451ce9b62156996aa3a67123f7060c906c8a8d104918c3f85dea4eec"}
Sep 18 19:50:45.135648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.135624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" event={"ID":"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920","Type":"ContainerStarted","Data":"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"}
Sep 18 19:50:45.135778 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.135667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" event={"ID":"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920","Type":"ContainerStarted","Data":"3c3c445fb094ed8b6ee927498d8c0f2570944e0505b6c3638059df2913b7ab3a"}
Sep 18 19:50:45.135778 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.135705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:50:45.160835 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.160786    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" podStartSLOduration=2.160768274 podStartE2EDuration="2.160768274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:45.157713545 +0000 UTC m=+2177.394840531" watchObservedRunningTime="2026-09-18 19:50:45.160768274 +0000 UTC m=+2177.397895261"
Sep 18 19:50:45.185411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.185369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podStartSLOduration=2.185355525 podStartE2EDuration="2.185355525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:45.18359574 +0000 UTC m=+2177.420722729" watchObservedRunningTime="2026-09-18 19:50:45.185355525 +0000 UTC m=+2177.422482510"
Sep 18 19:50:45.828587 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:45.828554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:53.829319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:53.829281    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:53.841710 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:53.841679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:50:54.193471 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:50:54.193380    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:51:16.143455 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:16.143424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:51:17.431523 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.431472    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:51:17.436376 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.436351    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:51:17.445343 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:51:17.445302    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:51:17.445343 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:51:17.445343 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:51:17.445343 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5" cmd=["/bin/sleep","15"]
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.445360    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" message="" err=<
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:51:17.445377    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerName="main" containerID="cri-o://26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"
Sep 18 19:51:17.445564 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.445418    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerName="main" containerID="cri-o://26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5" gracePeriod=60
Sep 18 19:51:17.787806 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.787774    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:51:17.878898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.878870    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.878943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.878966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.878991    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.879025    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879084 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.879056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw856\" (UniqueName: \"kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856\") pod \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\" (UID: \"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15\") "
Sep 18 19:51:17.879347 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.879270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:17.879347 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.879285    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home" (OuterVolumeSpecName: "home") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:17.879347 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.879293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:17.881471 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.881449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856" (OuterVolumeSpecName: "kube-api-access-jw856") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "kube-api-access-jw856". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:17.881798 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.881771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm" (OuterVolumeSpecName: "dshm") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:17.881891 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.881774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" (UID: "6ec6554e-1bf9-4e2a-a37c-d4fc41412a15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:17.979743 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979711    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:17.979866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979752    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:17.979866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979761    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:17.979866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979768    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:17.979866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979776    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:17.979866 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:17.979784    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw856\" (UniqueName: \"kubernetes.io/projected/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15-kube-api-access-jw856\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:51:18.308106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.308014    2582 generic.go:356] "Generic (PLEG): container finished" podID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerID="26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5" exitCode=0
Sep 18 19:51:18.308106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.308063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"
Sep 18 19:51:18.308319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.308065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" event={"ID":"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15","Type":"ContainerDied","Data":"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"}
Sep 18 19:51:18.308319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.308209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54" event={"ID":"6ec6554e-1bf9-4e2a-a37c-d4fc41412a15","Type":"ContainerDied","Data":"111058b4451ce9b62156996aa3a67123f7060c906c8a8d104918c3f85dea4eec"}
Sep 18 19:51:18.308319 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.308237    2582 scope.go:117] "RemoveContainer" containerID="26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"
Sep 18 19:51:18.317269 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.317249    2582 scope.go:117] "RemoveContainer" containerID="26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"
Sep 18 19:51:18.317533 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:51:18.317515    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5\": container with ID starting with 26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5 not found: ID does not exist" containerID="26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"
Sep 18 19:51:18.317598 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.317540    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5"} err="failed to get container status \"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5\": rpc error: code = NotFound desc = could not find container \"26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5\": container with ID starting with 26ea0f99bc4d8a6bfb7796c9b1d085b8f0167caf68882c8dd90208369efdc3a5 not found: ID does not exist"
Sep 18 19:51:18.324660 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.324640    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:51:18.330190 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:18.330170    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9952tk54"]
Sep 18 19:51:19.894455 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:19.894426    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" path="/var/lib/kubelet/pods/6ec6554e-1bf9-4e2a-a37c-d4fc41412a15/volumes"
Sep 18 19:51:26.142942 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:26.142901    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:32.452500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:32.452442    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main" containerID="cri-o://302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4" gracePeriod=45
Sep 18 19:51:36.142902 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:36.142865    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:46.143407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:46.143314    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:51:46.143845 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:46.143437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:51:56.143309 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:51:56.143273    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:02.697962 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.697922    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:52:02.813586 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.813561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs\") pod \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") "
Sep 18 19:52:02.813769 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.813657    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzxvd\" (UniqueName: \"kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd\") pod \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\" (UID: \"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920\") "
Sep 18 19:52:02.815850 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.815825    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd" (OuterVolumeSpecName: "kube-api-access-tzxvd") pod "c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" (UID: "c486b62e-ecb2-4dd9-a4b1-7d4b5d205920"). InnerVolumeSpecName "kube-api-access-tzxvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:02.815962 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.815868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" (UID: "c486b62e-ecb2-4dd9-a4b1-7d4b5d205920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:02.914718 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.914687    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:02.914718 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:02.914716    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxvd\" (UniqueName: \"kubernetes.io/projected/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920-kube-api-access-tzxvd\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:03.529612 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.529580    2582 generic.go:356] "Generic (PLEG): container finished" podID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerID="302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4" exitCode=0
Sep 18 19:52:03.529858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.529623    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"
Sep 18 19:52:03.529858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.529668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" event={"ID":"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920","Type":"ContainerDied","Data":"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"}
Sep 18 19:52:03.529858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.529718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66" event={"ID":"c486b62e-ecb2-4dd9-a4b1-7d4b5d205920","Type":"ContainerDied","Data":"3c3c445fb094ed8b6ee927498d8c0f2570944e0505b6c3638059df2913b7ab3a"}
Sep 18 19:52:03.529858 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.529757    2582 scope.go:117] "RemoveContainer" containerID="302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"
Sep 18 19:52:03.539116 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.539101    2582 scope.go:117] "RemoveContainer" containerID="302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"
Sep 18 19:52:03.539370 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:52:03.539348    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4\": container with ID starting with 302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4 not found: ID does not exist" containerID="302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"
Sep 18 19:52:03.539461 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.539376    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4"} err="failed to get container status \"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4\": rpc error: code = NotFound desc = could not find container \"302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4\": container with ID starting with 302b95f9f46160b49b8d041c279a35d4f0c32e9373688a882e951350dee959c4 not found: ID does not exist"
Sep 18 19:52:03.553965 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.553944    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:52:03.560276 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.560257    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek4l66"]
Sep 18 19:52:03.895100 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:03.895022    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" path="/var/lib/kubelet/pods/c486b62e-ecb2-4dd9-a4b1-7d4b5d205920/volumes"
Sep 18 19:52:10.790007 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:10.789971    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:52:10.808631 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:10.808601    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:52:10.817321 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:52:10.817286    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:52:10.817321 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:10.817321 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:10.817321 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269" cmd=["/bin/sleep","15"]
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:10.817338    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" message="" err=<
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:52:10.817352    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main" containerID="cri-o://f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"
Sep 18 19:52:10.817561 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:10.817383    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main" containerID="cri-o://f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269" gracePeriod=60
Sep 18 19:52:10.954410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:10.954378    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 19:52:11.159146 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.159120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:52:11.285829 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.285795    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.285852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.285886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.285922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.285948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286251 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286059    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:11.286251 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s97q7\" (UniqueName: \"kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7\") pod \"75185dce-79c6-4b3e-bab9-d239d6b635ed\" (UID: \"75185dce-79c6-4b3e-bab9-d239d6b635ed\") "
Sep 18 19:52:11.286251 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286223    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home" (OuterVolumeSpecName: "home") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:11.286410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286338    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.286410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286360    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.286510 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.286484    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:11.288358 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.288334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:11.288463 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.288413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm" (OuterVolumeSpecName: "dshm") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:11.288647 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.288630    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7" (OuterVolumeSpecName: "kube-api-access-s97q7") pod "75185dce-79c6-4b3e-bab9-d239d6b635ed" (UID: "75185dce-79c6-4b3e-bab9-d239d6b635ed"). InnerVolumeSpecName "kube-api-access-s97q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:11.387500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.387418    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.387500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.387448    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75185dce-79c6-4b3e-bab9-d239d6b635ed-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.387500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.387458    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75185dce-79c6-4b3e-bab9-d239d6b635ed-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.387500 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.387466    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s97q7\" (UniqueName: \"kubernetes.io/projected/75185dce-79c6-4b3e-bab9-d239d6b635ed-kube-api-access-s97q7\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:11.571207 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.571176    2582 generic.go:356] "Generic (PLEG): container finished" podID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerID="f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269" exitCode=0
Sep 18 19:52:11.571410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.571218    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"
Sep 18 19:52:11.571410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.571261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" event={"ID":"75185dce-79c6-4b3e-bab9-d239d6b635ed","Type":"ContainerDied","Data":"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"}
Sep 18 19:52:11.571410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.571320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br" event={"ID":"75185dce-79c6-4b3e-bab9-d239d6b635ed","Type":"ContainerDied","Data":"e4dee61eac452f90a6d7b7a8a7d9efbfe85f78f30bb43ea090abc7f3f46a61ba"}
Sep 18 19:52:11.571410 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.571354    2582 scope.go:117] "RemoveContainer" containerID="f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"
Sep 18 19:52:11.580963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.580947    2582 scope.go:117] "RemoveContainer" containerID="f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"
Sep 18 19:52:11.581211 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:52:11.581196    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269\": container with ID starting with f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269 not found: ID does not exist" containerID="f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"
Sep 18 19:52:11.581262 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.581220    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269"} err="failed to get container status \"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269\": rpc error: code = NotFound desc = could not find container \"f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269\": container with ID starting with f13d76bc1022105d519295e496bb811f4eccf7332afc4efc4294869b6628a269 not found: ID does not exist"
Sep 18 19:52:11.596886 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.596859    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:52:11.604748 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.604702    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-jl2br"]
Sep 18 19:52:11.894815 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:11.894789    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" path="/var/lib/kubelet/pods/75185dce-79c6-4b3e-bab9-d239d6b635ed/volumes"
Sep 18 19:52:19.918932 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:19.918898    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:25.810150 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:25.810110    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main" containerID="cri-o://1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368" gracePeriod=45
Sep 18 19:52:29.918393 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:29.918347    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:39.919149 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:39.919110    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:39.919602 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:39.919217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:52:49.918303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:49.918265    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:52:56.055715 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.055689    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:52:56.101707 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.101630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6sz\" (UniqueName: \"kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz\") pod \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") "
Sep 18 19:52:56.101707 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.101668    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs\") pod \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\" (UID: \"523296e3-9dc8-4f5a-a719-5f72d39f9e55\") "
Sep 18 19:52:56.103922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.103892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz" (OuterVolumeSpecName: "kube-api-access-nh6sz") pod "523296e3-9dc8-4f5a-a719-5f72d39f9e55" (UID: "523296e3-9dc8-4f5a-a719-5f72d39f9e55"). InnerVolumeSpecName "kube-api-access-nh6sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:56.104028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.103990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "523296e3-9dc8-4f5a-a719-5f72d39f9e55" (UID: "523296e3-9dc8-4f5a-a719-5f72d39f9e55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:56.202556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.202520    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6sz\" (UniqueName: \"kubernetes.io/projected/523296e3-9dc8-4f5a-a719-5f72d39f9e55-kube-api-access-nh6sz\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.202556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.202552    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523296e3-9dc8-4f5a-a719-5f72d39f9e55-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.786788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.786724    2582 generic.go:356] "Generic (PLEG): container finished" podID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerID="1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368" exitCode=0
Sep 18 19:52:56.786788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.786790    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"
Sep 18 19:52:56.787066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.786809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" event={"ID":"523296e3-9dc8-4f5a-a719-5f72d39f9e55","Type":"ContainerDied","Data":"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"}
Sep 18 19:52:56.787066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.786851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47" event={"ID":"523296e3-9dc8-4f5a-a719-5f72d39f9e55","Type":"ContainerDied","Data":"c31af23e0ba3d341e7d7816f32af9c782583d6cdd00b09807dff1881fafa4c50"}
Sep 18 19:52:56.787066 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.786872    2582 scope.go:117] "RemoveContainer" containerID="1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"
Sep 18 19:52:56.796218 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.796202    2582 scope.go:117] "RemoveContainer" containerID="1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"
Sep 18 19:52:56.796507 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:52:56.796485    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368\": container with ID starting with 1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368 not found: ID does not exist" containerID="1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"
Sep 18 19:52:56.796556 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.796515    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368"} err="failed to get container status \"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368\": rpc error: code = NotFound desc = could not find container \"1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368\": container with ID starting with 1fc01542bbc0df195cf809ad51e77a180320e846620bd8b5e84c8ce04876d368 not found: ID does not exist"
Sep 18 19:52:56.813459 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.813431    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:52:56.815692 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:56.815669    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schem5m47"]
Sep 18 19:52:57.895377 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:52:57.895342    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" path="/var/lib/kubelet/pods/523296e3-9dc8-4f5a-a719-5f72d39f9e55/volumes"
Sep 18 19:53:09.430193 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430105    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:53:09.430633 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430613    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main"
Sep 18 19:53:09.430685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430668    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main"
Sep 18 19:53:09.430685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430681    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerName="main"
Sep 18 19:53:09.430788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430691    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerName="main"
Sep 18 19:53:09.430788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430709    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main"
Sep 18 19:53:09.430788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430717    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main"
Sep 18 19:53:09.430788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430751    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main"
Sep 18 19:53:09.430788 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430759    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main"
Sep 18 19:53:09.430948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430860    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="523296e3-9dc8-4f5a-a719-5f72d39f9e55" containerName="main"
Sep 18 19:53:09.430948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430877    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75185dce-79c6-4b3e-bab9-d239d6b635ed" containerName="main"
Sep 18 19:53:09.430948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430887    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c486b62e-ecb2-4dd9-a4b1-7d4b5d205920" containerName="main"
Sep 18 19:53:09.430948 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.430895    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec6554e-1bf9-4e2a-a37c-d4fc41412a15" containerName="main"
Sep 18 19:53:09.435381 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.435361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.442297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.442278    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 19:53:09.443249 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.443233    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ltr9b\""
Sep 18 19:53:09.443303 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.443248    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 19:53:09.443376 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.443353    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 19:53:09.455888 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.455863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:53:09.491118 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.491297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.491297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.491297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.491297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.491297 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.491287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6dd\" (UniqueName: \"kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592120 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592397 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592442 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6dd\" (UniqueName: \"kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592637 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.592803 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.592680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.594579 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.594558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.594785 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.594767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.603959 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.603938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6dd\" (UniqueName: \"kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:09.745263 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:09.745222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:10.033827 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:10.033801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:53:10.034827 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:53:10.034804    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f34749_0619_454a_95c4_236ec7492a9c.slice/crio-1d3142b6d0943d51311ba6717d7cec6d6c042ecc771bb2c56439b798856c2cee WatchSource:0}: Error finding container 1d3142b6d0943d51311ba6717d7cec6d6c042ecc771bb2c56439b798856c2cee: Status 404 returned error can't find the container with id 1d3142b6d0943d51311ba6717d7cec6d6c042ecc771bb2c56439b798856c2cee
Sep 18 19:53:10.855147 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:10.855099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" event={"ID":"04f34749-0619-454a-95c4-236ec7492a9c","Type":"ContainerStarted","Data":"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"}
Sep 18 19:53:10.855563 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:10.855159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" event={"ID":"04f34749-0619-454a-95c4-236ec7492a9c","Type":"ContainerStarted","Data":"1d3142b6d0943d51311ba6717d7cec6d6c042ecc771bb2c56439b798856c2cee"}
Sep 18 19:53:10.893826 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:10.893774    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" podStartSLOduration=1.893757641 podStartE2EDuration="1.893757641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:10.893003015 +0000 UTC m=+2323.130130001" watchObservedRunningTime="2026-09-18 19:53:10.893757641 +0000 UTC m=+2323.130884623"
Sep 18 19:53:11.745487 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:11.745449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:19.745458 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:19.745422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:19.758033 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:19.758009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:19.907979 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:19.907944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:53:27.180024 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.179991    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:53:27.183106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.183087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.187173 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.187148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-697gw\""
Sep 18 19:53:27.187292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.187179    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 19:53:27.187292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.187151    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 19:53:27.187292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.187148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-bxlj5\""
Sep 18 19:53:27.187292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.187156    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 19:53:27.197794 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.197772    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:53:27.227717 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.227683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.227901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.227769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwhb\" (UniqueName: \"kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.329076 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.329021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.329252 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.329085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwhb\" (UniqueName: \"kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.331838 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.331814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.338849 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.338826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwhb\" (UniqueName: \"kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.493179 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.493142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.794440 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.794411    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:53:27.796644 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:53:27.796616    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7d3508_2363_4fb6_b136_15d9ed543bb6.slice/crio-39f883b91b1a5b477ed2f34ca81488c2a0225c01b02938d8e0ceb15097e22f8f WatchSource:0}: Error finding container 39f883b91b1a5b477ed2f34ca81488c2a0225c01b02938d8e0ceb15097e22f8f: Status 404 returned error can't find the container with id 39f883b91b1a5b477ed2f34ca81488c2a0225c01b02938d8e0ceb15097e22f8f
Sep 18 19:53:27.938318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.938278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" event={"ID":"1a7d3508-2363-4fb6-b136-15d9ed543bb6","Type":"ContainerStarted","Data":"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"}
Sep 18 19:53:27.938806 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.938780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:53:27.939008 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.938989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" event={"ID":"1a7d3508-2363-4fb6-b136-15d9ed543bb6","Type":"ContainerStarted","Data":"39f883b91b1a5b477ed2f34ca81488c2a0225c01b02938d8e0ceb15097e22f8f"}
Sep 18 19:53:27.965955 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:27.965895    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podStartSLOduration=0.965874631 podStartE2EDuration="965.874631ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:27.963402027 +0000 UTC m=+2340.200529014" watchObservedRunningTime="2026-09-18 19:53:27.965874631 +0000 UTC m=+2340.203001617"
Sep 18 19:53:59.950265 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:53:59.950239    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:54:29.973034 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:54:29.973008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:54:29.980118 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:54:29.980095    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:54:29.980259 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:54:29.980160    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:54:29.986922 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:54:29.986905    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:55:26.829250 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:26.829219    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:55:26.840326 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:55:26.840293    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:55:26.840326 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:26.840326 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:26.840326 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26" cmd=["/bin/sleep","15"]
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:26.840344    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" message="" err=<
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:55:26.840359    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main" containerID="cri-o://dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"
Sep 18 19:55:26.840593 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:26.840412    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main" containerID="cri-o://dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26" gracePeriod=60
Sep 18 19:55:26.897146 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:26.897113    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 19:55:27.199651 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.199625    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:55:27.334318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp6dd\" (UniqueName: \"kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334422    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334490    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334555 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm\") pod \"04f34749-0619-454a-95c4-236ec7492a9c\" (UID: \"04f34749-0619-454a-95c4-236ec7492a9c\") "
Sep 18 19:55:27.334825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home" (OuterVolumeSpecName: "home") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.334825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334754    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.334825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.334974 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.334846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.336763 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.336710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm" (OuterVolumeSpecName: "dshm") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:27.336763 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.336721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:27.336911 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.336765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd" (OuterVolumeSpecName: "kube-api-access-jp6dd") pod "04f34749-0619-454a-95c4-236ec7492a9c" (UID: "04f34749-0619-454a-95c4-236ec7492a9c"). InnerVolumeSpecName "kube-api-access-jp6dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:27.436132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.436062    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.436132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.436097    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.436132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.436110    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f34749-0619-454a-95c4-236ec7492a9c-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.436132 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.436122    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f34749-0619-454a-95c4-236ec7492a9c-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.436354 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.436134    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6dd\" (UniqueName: \"kubernetes.io/projected/04f34749-0619-454a-95c4-236ec7492a9c-kube-api-access-jp6dd\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:55:27.515231 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.515202    2582 generic.go:356] "Generic (PLEG): container finished" podID="04f34749-0619-454a-95c4-236ec7492a9c" containerID="dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26" exitCode=0
Sep 18 19:55:27.515378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.515244    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"
Sep 18 19:55:27.515378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.515287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" event={"ID":"04f34749-0619-454a-95c4-236ec7492a9c","Type":"ContainerDied","Data":"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"}
Sep 18 19:55:27.515378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.515337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll" event={"ID":"04f34749-0619-454a-95c4-236ec7492a9c","Type":"ContainerDied","Data":"1d3142b6d0943d51311ba6717d7cec6d6c042ecc771bb2c56439b798856c2cee"}
Sep 18 19:55:27.515378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.515365    2582 scope.go:117] "RemoveContainer" containerID="dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"
Sep 18 19:55:27.524636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.524621    2582 scope.go:117] "RemoveContainer" containerID="dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"
Sep 18 19:55:27.524921 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:55:27.524903    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26\": container with ID starting with dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26 not found: ID does not exist" containerID="dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"
Sep 18 19:55:27.524986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.524929    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26"} err="failed to get container status \"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26\": rpc error: code = NotFound desc = could not find container \"dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26\": container with ID starting with dc35a628092fd2c93b8468d7a39adb9305192d96e92fb2887a412d7a1dc58a26 not found: ID does not exist"
Sep 18 19:55:27.565090 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.565059    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:55:27.579841 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.579810    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vtwll"]
Sep 18 19:55:27.895697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:27.895657    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f34749-0619-454a-95c4-236ec7492a9c" path="/var/lib/kubelet/pods/04f34749-0619-454a-95c4-236ec7492a9c/volumes"
Sep 18 19:55:33.072791 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:33.072749    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:55:39.950056 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:39.950016    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:48.091822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:48.091755    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main" containerID="cri-o://7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec" gracePeriod=45
Sep 18 19:55:49.950106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:49.950061    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:59.950920 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:59.950882    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:55:59.951457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:55:59.950994    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:56:09.950364 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:09.950278    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:56:18.324963 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.324934    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:56:18.413002 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.412923    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs\") pod \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") "
Sep 18 19:56:18.413002 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.412978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nwhb\" (UniqueName: \"kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb\") pod \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\" (UID: \"1a7d3508-2363-4fb6-b136-15d9ed543bb6\") "
Sep 18 19:56:18.415293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.415257    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a7d3508-2363-4fb6-b136-15d9ed543bb6" (UID: "1a7d3508-2363-4fb6-b136-15d9ed543bb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:18.415293 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.415264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb" (OuterVolumeSpecName: "kube-api-access-7nwhb") pod "1a7d3508-2363-4fb6-b136-15d9ed543bb6" (UID: "1a7d3508-2363-4fb6-b136-15d9ed543bb6"). InnerVolumeSpecName "kube-api-access-7nwhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:18.513820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.513782    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7d3508-2363-4fb6-b136-15d9ed543bb6-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:56:18.513820 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.513817    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwhb\" (UniqueName: \"kubernetes.io/projected/1a7d3508-2363-4fb6-b136-15d9ed543bb6-kube-api-access-7nwhb\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:56:18.765821 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.765778    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerID="7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec" exitCode=0
Sep 18 19:56:18.765986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.765828    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"
Sep 18 19:56:18.765986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.765846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" event={"ID":"1a7d3508-2363-4fb6-b136-15d9ed543bb6","Type":"ContainerDied","Data":"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"}
Sep 18 19:56:18.765986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.765897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br" event={"ID":"1a7d3508-2363-4fb6-b136-15d9ed543bb6","Type":"ContainerDied","Data":"39f883b91b1a5b477ed2f34ca81488c2a0225c01b02938d8e0ceb15097e22f8f"}
Sep 18 19:56:18.765986 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.765926    2582 scope.go:117] "RemoveContainer" containerID="7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"
Sep 18 19:56:18.774885 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.774863    2582 scope.go:117] "RemoveContainer" containerID="7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"
Sep 18 19:56:18.775167 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:56:18.775147    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec\": container with ID starting with 7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec not found: ID does not exist" containerID="7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"
Sep 18 19:56:18.775217 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.775175    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec"} err="failed to get container status \"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec\": rpc error: code = NotFound desc = could not find container \"7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec\": container with ID starting with 7d54c740616829e21bc5d1caa3a082571ac6b2436bb7d0c7a7b1425a9fc5ecec not found: ID does not exist"
Sep 18 19:56:18.792903 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.792879    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:56:18.802559 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:18.802534    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schew57br"]
Sep 18 19:56:19.894583 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:19.894550    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" path="/var/lib/kubelet/pods/1a7d3508-2363-4fb6-b136-15d9ed543bb6/volumes"
Sep 18 19:56:24.140545 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140509    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main"
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140814    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main"
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140829    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main"
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140835    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main"
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140880    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7d3508-2363-4fb6-b136-15d9ed543bb6" containerName="main"
Sep 18 19:56:24.141018 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.140891    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f34749-0619-454a-95c4-236ec7492a9c" containerName="main"
Sep 18 19:56:24.143869 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.143852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.148360 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.148329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 19:56:24.148527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.148502    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 19:56:24.148527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.148517    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:56:24.148670 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.148511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 19:56:24.148670 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.148544    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wws8w\""
Sep 18 19:56:24.149055 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.149041    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 18 19:56:24.170493 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.170470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:56:24.252541 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252682 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k828l\" (UniqueName: \"kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.252944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.252841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353489 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k828l\" (UniqueName: \"kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353636 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.353969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.353957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.354141 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.354016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.354182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.354137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.354182 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.354137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.354425 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.354407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.356180 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.356157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.356454 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.356434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.365494 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.365467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k828l\" (UniqueName: \"kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.454160 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.454083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:24.763027 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.762995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:56:24.763316 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:56:24.763295    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941e8f39_5701_4933_a6c9_f5a0176583b1.slice/crio-60d1445ba1dca2e6e8a1c928be7c37a20b5cd08ef41f361ce8b442be3cd11df0 WatchSource:0}: Error finding container 60d1445ba1dca2e6e8a1c928be7c37a20b5cd08ef41f361ce8b442be3cd11df0: Status 404 returned error can't find the container with id 60d1445ba1dca2e6e8a1c928be7c37a20b5cd08ef41f361ce8b442be3cd11df0
Sep 18 19:56:24.801872 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:24.801844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerStarted","Data":"60d1445ba1dca2e6e8a1c928be7c37a20b5cd08ef41f361ce8b442be3cd11df0"}
Sep 18 19:56:25.809452 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:25.809414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerStarted","Data":"2cf6dcf4977aad948e8106ebef72c91451ee05744177111222df4312608d4b7c"}
Sep 18 19:56:27.820712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:27.820630    2582 generic.go:356] "Generic (PLEG): container finished" podID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerID="2cf6dcf4977aad948e8106ebef72c91451ee05744177111222df4312608d4b7c" exitCode=0
Sep 18 19:56:27.820712 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:27.820682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerDied","Data":"2cf6dcf4977aad948e8106ebef72c91451ee05744177111222df4312608d4b7c"}
Sep 18 19:56:28.828052 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:28.828015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerStarted","Data":"67c62419f910b89f7c30c9a51ffe42f11807b822b447eb84b49654c8a98d1726"}
Sep 18 19:56:28.858035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:28.857978    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podStartSLOduration=2.20796277 podStartE2EDuration="4.857961906s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.649999136s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:28.854989809 +0000 UTC m=+2521.092116794" watchObservedRunningTime="2026-09-18 19:56:28.857961906 +0000 UTC m=+2521.095088891"
Sep 18 19:56:29.454378 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:29.454341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:34.454535 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:34.454500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:56:34.456357 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:34.456333    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:56:44.454796 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:44.454755    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:56:45.690625 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.690577    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:56:45.695100 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.695076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.697969 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.697943    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-bvxgk\""
Sep 18 19:56:45.698865 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.698840    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 19:56:45.698993 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.698934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-r5kzs\""
Sep 18 19:56:45.699103 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.699085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 19:56:45.699229 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.699208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 19:56:45.708389 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.708365    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:56:45.842440 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.842390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.842652 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.842453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6hl\" (UniqueName: \"kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.942929 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.942827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.942929 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.942894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6hl\" (UniqueName: \"kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.945745 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.945700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:45.953260 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:45.953232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6hl\" (UniqueName: \"kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:46.005855 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.005815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:46.289115 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.289087    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:56:46.290995 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:56:46.290944    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6c9322_12a2_4ec4_bfcd_b534001e4d18.slice/crio-44edc32447ad51d8166d0cf16efa7fb0f6f4d12f912dc634e026b00c0b11dfa9 WatchSource:0}: Error finding container 44edc32447ad51d8166d0cf16efa7fb0f6f4d12f912dc634e026b00c0b11dfa9: Status 404 returned error can't find the container with id 44edc32447ad51d8166d0cf16efa7fb0f6f4d12f912dc634e026b00c0b11dfa9
Sep 18 19:56:46.915898 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.915859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" event={"ID":"8b6c9322-12a2-4ec4-bfcd-b534001e4d18","Type":"ContainerStarted","Data":"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"}
Sep 18 19:56:46.916325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.915916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" event={"ID":"8b6c9322-12a2-4ec4-bfcd-b534001e4d18","Type":"ContainerStarted","Data":"44edc32447ad51d8166d0cf16efa7fb0f6f4d12f912dc634e026b00c0b11dfa9"}
Sep 18 19:56:46.916325 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.915954    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:56:46.953479 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:46.953424    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podStartSLOduration=1.95340726 podStartE2EDuration="1.95340726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:46.95275938 +0000 UTC m=+2539.189886374" watchObservedRunningTime="2026-09-18 19:56:46.95340726 +0000 UTC m=+2539.190534280"
Sep 18 19:56:54.455051 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:56:54.455001    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:57:04.455073 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:04.455022    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:57:14.454980 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:14.454919    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:57:17.923868 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:17.923836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:57:24.454715 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:24.454670    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:57:34.454590 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:34.454543    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 18 19:57:44.464233 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:44.464158    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:57:44.471901 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:57:44.471875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:58:12.817264 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:12.817226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:58:27.840411 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:27.840364    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" containerID="cri-o://67c62419f910b89f7c30c9a51ffe42f11807b822b447eb84b49654c8a98d1726" gracePeriod=45
Sep 18 19:58:28.440334 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.440298    2582 generic.go:356] "Generic (PLEG): container finished" podID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerID="67c62419f910b89f7c30c9a51ffe42f11807b822b447eb84b49654c8a98d1726" exitCode=0
Sep 18 19:58:28.440508 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.440339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerDied","Data":"67c62419f910b89f7c30c9a51ffe42f11807b822b447eb84b49654c8a98d1726"}
Sep 18 19:58:28.483642 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.483617    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:58:28.547028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.546938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.546990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547028 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k828l\" (UniqueName: \"kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547038    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547053    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547148    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547318 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547189    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home\") pod \"941e8f39-5701-4933-a6c9-f5a0176583b1\" (UID: \"941e8f39-5701-4933-a6c9-f5a0176583b1\") "
Sep 18 19:58:28.547575 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547393    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:28.547648 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547618    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:28.547744 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.547707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home" (OuterVolumeSpecName: "home") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:28.549831 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.549806    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm" (OuterVolumeSpecName: "dshm") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:28.550031 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.550001    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l" (OuterVolumeSpecName: "kube-api-access-k828l") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "kube-api-access-k828l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:28.550097 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.550082    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:28.559741 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.559692    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:28.571841 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.571815    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "941e8f39-5701-4933-a6c9-f5a0176583b1" (UID: "941e8f39-5701-4933-a6c9-f5a0176583b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:28.647944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647913    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k828l\" (UniqueName: \"kubernetes.io/projected/941e8f39-5701-4933-a6c9-f5a0176583b1-kube-api-access-k828l\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.647944 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647940    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e8f39-5701-4933-a6c9-f5a0176583b1-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647974    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647982    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647991    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/941e8f39-5701-4933-a6c9-f5a0176583b1-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.647998    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.648006    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:28.648106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:28.648013    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e8f39-5701-4933-a6c9-f5a0176583b1-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:58:29.447012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.446982    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"
Sep 18 19:58:29.447419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.446981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" event={"ID":"941e8f39-5701-4933-a6c9-f5a0176583b1","Type":"ContainerDied","Data":"60d1445ba1dca2e6e8a1c928be7c37a20b5cd08ef41f361ce8b442be3cd11df0"}
Sep 18 19:58:29.447419 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.447127    2582 scope.go:117] "RemoveContainer" containerID="67c62419f910b89f7c30c9a51ffe42f11807b822b447eb84b49654c8a98d1726"
Sep 18 19:58:29.462597 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.462576    2582 scope.go:117] "RemoveContainer" containerID="2cf6dcf4977aad948e8106ebef72c91451ee05744177111222df4312608d4b7c"
Sep 18 19:58:29.465035 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.465007    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:58:29.482106 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.482084    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:58:29.492015 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.491995    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c459fb75-blbnc"]
Sep 18 19:58:29.895012 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:29.894982    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" path="/var/lib/kubelet/pods/941e8f39-5701-4933-a6c9-f5a0176583b1/volumes"
Sep 18 19:58:50.101436 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:50.101401    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:58:57.924025 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:58:57.923990    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:05.121162 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:05.121117    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main" containerID="cri-o://68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473" gracePeriod=45
Sep 18 19:59:07.923407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:07.923371    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:17.924253 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:17.924165    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:17.924685 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:17.924287    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:59:27.923994 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:27.923958    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 19:59:30.006316 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:30.006288    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:59:30.013063 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:30.013039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:59:30.013247 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:30.013230    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 19:59:30.020136 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:30.020109    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 19:59:35.366261 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.366238    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:59:35.434787 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.434707    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs\") pod \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") "
Sep 18 19:59:35.434787 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.434776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n6hl\" (UniqueName: \"kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl\") pod \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\" (UID: \"8b6c9322-12a2-4ec4-bfcd-b534001e4d18\") "
Sep 18 19:59:35.437114 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.437082    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b6c9322-12a2-4ec4-bfcd-b534001e4d18" (UID: "8b6c9322-12a2-4ec4-bfcd-b534001e4d18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:35.437114 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.437091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl" (OuterVolumeSpecName: "kube-api-access-5n6hl") pod "8b6c9322-12a2-4ec4-bfcd-b534001e4d18" (UID: "8b6c9322-12a2-4ec4-bfcd-b534001e4d18"). InnerVolumeSpecName "kube-api-access-5n6hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:35.535695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.535666    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:59:35.535695 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.535692    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6hl\" (UniqueName: \"kubernetes.io/projected/8b6c9322-12a2-4ec4-bfcd-b534001e4d18-kube-api-access-5n6hl\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 19:59:35.750110 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.750082    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerID="68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473" exitCode=0
Sep 18 19:59:35.750243 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.750120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"
Sep 18 19:59:35.750243 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.750172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" event={"ID":"8b6c9322-12a2-4ec4-bfcd-b534001e4d18","Type":"ContainerDied","Data":"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"}
Sep 18 19:59:35.750243 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.750229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf" event={"ID":"8b6c9322-12a2-4ec4-bfcd-b534001e4d18","Type":"ContainerDied","Data":"44edc32447ad51d8166d0cf16efa7fb0f6f4d12f912dc634e026b00c0b11dfa9"}
Sep 18 19:59:35.750413 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.750258    2582 scope.go:117] "RemoveContainer" containerID="68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"
Sep 18 19:59:35.760089 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.760073    2582 scope.go:117] "RemoveContainer" containerID="68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"
Sep 18 19:59:35.760337 ip-10-0-143-75 kubenswrapper[2582]: E0918 19:59:35.760318    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473\": container with ID starting with 68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473 not found: ID does not exist" containerID="68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"
Sep 18 19:59:35.760407 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.760342    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473"} err="failed to get container status \"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473\": rpc error: code = NotFound desc = could not find container \"68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473\": container with ID starting with 68a75126f60b8c4528c64d7dcf1fc8c957b2ccc763f26706ad0c65161047d473 not found: ID does not exist"
Sep 18 19:59:35.785174 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.785152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:59:35.803837 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.803814    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche926tf"]
Sep 18 19:59:35.894429 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:35.894401    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" path="/var/lib/kubelet/pods/8b6c9322-12a2-4ec4-bfcd-b534001e4d18/volumes"
Sep 18 19:59:48.406862 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.406828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407119    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407130    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407143    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407148    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407159    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="storage-initializer"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407165    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="storage-initializer"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407212    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="941e8f39-5701-4933-a6c9-f5a0176583b1" containerName="main"
Sep 18 19:59:48.407351 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.407222    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6c9322-12a2-4ec4-bfcd-b534001e4d18" containerName="main"
Sep 18 19:59:48.411515 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.411498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.414436 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.414417    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 18 19:59:48.414557 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.414538    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:59:48.414825 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.414808    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-gxg4t\""
Sep 18 19:59:48.414892 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.414875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 19:59:48.415335 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.415322    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 19:59:48.415790 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.415777    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 19:59:48.425371 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.425348    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 19:59:48.528394 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528558 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.528697 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.528652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629722 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.629992 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.629912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.630306 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.630289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.630366 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.630317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.630366 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.630350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.630477 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.630454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.630675 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.630657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.632217 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.632197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.632583 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.632562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.640448 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.640422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn\") pod \"custom-route-timeout-test-kserve-9ffbb77b8-vfm99\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.687665 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.687599    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 19:59:48.692127 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.692109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.696771 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.696751    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-fntzr\""
Sep 18 19:59:48.709811 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.709789    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 19:59:48.721743 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.721708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:48.831384 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.831349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxwb\" (UniqueName: \"kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.831527 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.831389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.932572 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.932543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.932706 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.932623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxwb\" (UniqueName: \"kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.935213 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.935194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:48.945400 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:48.945350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxwb\" (UniqueName: \"kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:49.002292 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.002265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:49.032457 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.032431    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 19:59:49.034190 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:59:49.034161    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8befa54a_cb9f_4586_a3f8_80cfdbd94ba7.slice/crio-33dd86b75025846e1a234b96a87d883ee5784625de70158572880f91683666aa WatchSource:0}: Error finding container 33dd86b75025846e1a234b96a87d883ee5784625de70158572880f91683666aa: Status 404 returned error can't find the container with id 33dd86b75025846e1a234b96a87d883ee5784625de70158572880f91683666aa
Sep 18 19:59:49.294045 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.294000    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 19:59:49.294203 ip-10-0-143-75 kubenswrapper[2582]: W0918 19:59:49.294176    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb784b79_0534_4454_b83e_50282c7bd3e1.slice/crio-90e9bcb730ee3e0375bbe2b2a703d12ef8d860ace649c3c39464541ac162449d WatchSource:0}: Error finding container 90e9bcb730ee3e0375bbe2b2a703d12ef8d860ace649c3c39464541ac162449d: Status 404 returned error can't find the container with id 90e9bcb730ee3e0375bbe2b2a703d12ef8d860ace649c3c39464541ac162449d
Sep 18 19:59:49.816576 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.816538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" event={"ID":"cb784b79-0534-4454-b83e-50282c7bd3e1","Type":"ContainerStarted","Data":"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"}
Sep 18 19:59:49.817051 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.816593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" event={"ID":"cb784b79-0534-4454-b83e-50282c7bd3e1","Type":"ContainerStarted","Data":"90e9bcb730ee3e0375bbe2b2a703d12ef8d860ace649c3c39464541ac162449d"}
Sep 18 19:59:49.817051 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.816636    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 19:59:49.818280 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.818257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerStarted","Data":"61bfbc330992f5471fb12fcabf26d4a1d50f29256591eeb998073d32bee0699a"}
Sep 18 19:59:49.818386 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.818295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerStarted","Data":"33dd86b75025846e1a234b96a87d883ee5784625de70158572880f91683666aa"}
Sep 18 19:59:49.837334 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:49.837274    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podStartSLOduration=1.837248512 podStartE2EDuration="1.837248512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:49.837130957 +0000 UTC m=+2722.074257944" watchObservedRunningTime="2026-09-18 19:59:49.837248512 +0000 UTC m=+2722.074375499"
Sep 18 19:59:51.829017 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:51.828986    2582 generic.go:356] "Generic (PLEG): container finished" podID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerID="61bfbc330992f5471fb12fcabf26d4a1d50f29256591eeb998073d32bee0699a" exitCode=0
Sep 18 19:59:51.829376 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:51.829039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerDied","Data":"61bfbc330992f5471fb12fcabf26d4a1d50f29256591eeb998073d32bee0699a"}
Sep 18 19:59:52.836996 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:52.836964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerStarted","Data":"0b0cabe2608a52e417a5258d7c0766975a24bed01495d74c1bfb67a5027ede29"}
Sep 18 19:59:52.872190 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:52.872137    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podStartSLOduration=2.6430423530000002 podStartE2EDuration="4.872122037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.229079684s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:52.869435956 +0000 UTC m=+2725.106562941" watchObservedRunningTime="2026-09-18 19:59:52.872122037 +0000 UTC m=+2725.109249022"
Sep 18 19:59:53.722822 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:53.722787    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:58.722851 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:58.722808    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 19:59:58.724632 ip-10-0-143-75 kubenswrapper[2582]: I0918 19:59:58.724601    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:08.722525 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:08.722463    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:18.722652 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:18.722612    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:20.824900 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:20.824867    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 20:00:28.723169 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:28.723125    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:38.722375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:38.722325    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:48.723190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:48.723139    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:00:58.722874 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:00:58.722832    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:01:08.723094 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:08.723051    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:01:18.723045 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:18.722997    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:01:28.722420 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:28.722373    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:01:38.722427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:38.722377    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 18 20:01:48.732481 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:48.732446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 20:01:48.740190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:48.740155    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 20:01:53.822059 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:53.822010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 20:01:53.825120 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:01:53.825086    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 20:02:00.824723 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:00.824686    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:08.849004 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:08.848954    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main" containerID="cri-o://0b0cabe2608a52e417a5258d7c0766975a24bed01495d74c1bfb67a5027ede29" gracePeriod=45
Sep 18 20:02:08.850146 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:08.850116    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main" containerID="cri-o://1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7" gracePeriod=45
Sep 18 20:02:09.545147 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.545117    2582 generic.go:356] "Generic (PLEG): container finished" podID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerID="0b0cabe2608a52e417a5258d7c0766975a24bed01495d74c1bfb67a5027ede29" exitCode=0
Sep 18 20:02:09.545291 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.545185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerDied","Data":"0b0cabe2608a52e417a5258d7c0766975a24bed01495d74c1bfb67a5027ede29"}
Sep 18 20:02:09.585088 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.585065    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 20:02:09.682135 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682135 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682173    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache\") pod \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\" (UID: \"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7\") "
Sep 18 20:02:09.682836 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682669    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:09.682836 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.682682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:09.683469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.683440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home" (OuterVolumeSpecName: "home") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:09.684797 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.684773    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm" (OuterVolumeSpecName: "dshm") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:09.684900 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.684853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:09.684948 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.684894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn" (OuterVolumeSpecName: "kube-api-access-5bnsn") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "kube-api-access-5bnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:09.700857 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.700831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:09.708511 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.708489    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" (UID: "8befa54a-cb9f-4586-a3f8-80cfdbd94ba7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:09.782763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782740    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782764    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782776    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kube-api-access-5bnsn\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782784    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782792    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782799    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782806    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:09.782885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:09.782813    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:10.555008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.554960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99" event={"ID":"8befa54a-cb9f-4586-a3f8-80cfdbd94ba7","Type":"ContainerDied","Data":"33dd86b75025846e1a234b96a87d883ee5784625de70158572880f91683666aa"}
Sep 18 20:02:10.555008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.555000    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"
Sep 18 20:02:10.555485 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.555056    2582 scope.go:117] "RemoveContainer" containerID="0b0cabe2608a52e417a5258d7c0766975a24bed01495d74c1bfb67a5027ede29"
Sep 18 20:02:10.563832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.563812    2582 scope.go:117] "RemoveContainer" containerID="61bfbc330992f5471fb12fcabf26d4a1d50f29256591eeb998073d32bee0699a"
Sep 18 20:02:10.575666 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.575644    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 20:02:10.580882 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.580863    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9ffbb77b8-vfm99"]
Sep 18 20:02:10.824545 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:10.824454    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:11.894662 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:11.894626    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" path="/var/lib/kubelet/pods/8befa54a-cb9f-4586-a3f8-80cfdbd94ba7/volumes"
Sep 18 20:02:20.825393 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:20.825356    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:20.825842 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:20.825462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 20:02:30.825008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:30.824968    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:39.095149 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.095123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 20:02:39.190323 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.190229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrxwb\" (UniqueName: \"kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb\") pod \"cb784b79-0534-4454-b83e-50282c7bd3e1\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") "
Sep 18 20:02:39.190323 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.190273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs\") pod \"cb784b79-0534-4454-b83e-50282c7bd3e1\" (UID: \"cb784b79-0534-4454-b83e-50282c7bd3e1\") "
Sep 18 20:02:39.192561 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.192528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb784b79-0534-4454-b83e-50282c7bd3e1" (UID: "cb784b79-0534-4454-b83e-50282c7bd3e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:39.192675 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.192606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb" (OuterVolumeSpecName: "kube-api-access-mrxwb") pod "cb784b79-0534-4454-b83e-50282c7bd3e1" (UID: "cb784b79-0534-4454-b83e-50282c7bd3e1"). InnerVolumeSpecName "kube-api-access-mrxwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:39.291375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.291340    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxwb\" (UniqueName: \"kubernetes.io/projected/cb784b79-0534-4454-b83e-50282c7bd3e1-kube-api-access-mrxwb\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:39.291375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.291369    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb784b79-0534-4454-b83e-50282c7bd3e1-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:02:39.692401 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.692367    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerID="1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7" exitCode=0
Sep 18 20:02:39.692571 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.692409    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"
Sep 18 20:02:39.692571 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.692456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" event={"ID":"cb784b79-0534-4454-b83e-50282c7bd3e1","Type":"ContainerDied","Data":"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"}
Sep 18 20:02:39.692571 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.692532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg" event={"ID":"cb784b79-0534-4454-b83e-50282c7bd3e1","Type":"ContainerDied","Data":"90e9bcb730ee3e0375bbe2b2a703d12ef8d860ace649c3c39464541ac162449d"}
Sep 18 20:02:39.692571 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.692554    2582 scope.go:117] "RemoveContainer" containerID="1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"
Sep 18 20:02:39.704487 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.704471    2582 scope.go:117] "RemoveContainer" containerID="1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"
Sep 18 20:02:39.704818 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:02:39.704798    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7\": container with ID starting with 1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7 not found: ID does not exist" containerID="1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"
Sep 18 20:02:39.704892 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.704826    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7"} err="failed to get container status \"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7\": rpc error: code = NotFound desc = could not find container \"1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7\": container with ID starting with 1aacff712f0069349c48bf21a50511c51682f67fe9102faf1f49e8e20f3802c7 not found: ID does not exist"
Sep 18 20:02:39.716014 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.715991    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 20:02:39.722229 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.722208    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc588m6gg"]
Sep 18 20:02:39.894830 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:39.894798    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" path="/var/lib/kubelet/pods/cb784b79-0534-4454-b83e-50282c7bd3e1/volumes"
Sep 18 20:02:55.192047 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192014    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192289    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192299    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192309    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192314    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192323    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="storage-initializer"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="storage-initializer"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192377    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8befa54a-cb9f-4586-a3f8-80cfdbd94ba7" containerName="main"
Sep 18 20:02:55.192523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.192385    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb784b79-0534-4454-b83e-50282c7bd3e1" containerName="main"
Sep 18 20:02:55.195218 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.195198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.198678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.198653    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:02:55.199164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.199141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 18 20:02:55.199272 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.199227    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 20:02:55.199628 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.199608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 20:02:55.200072 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.200057    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-2s75k\""
Sep 18 20:02:55.200147 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.200066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 20:02:55.210316 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.210296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:02:55.315267 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315410 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315410 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315410 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkgf\" (UniqueName: \"kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315410 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.315611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.315493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416214 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416214 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkgf\" (UniqueName: \"kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416682 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416682 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416682 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416682 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416965 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.416965 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.416771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.417090 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.417073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.418880 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.418860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.419173 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.419156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.425956 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.425937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkgf\" (UniqueName: \"kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf\") pod \"router-with-refs-test-kserve-85bc676c96-6jt8c\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.453417 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.453345    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:02:55.456472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.456453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.458895 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.458873    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-mb2vq\""
Sep 18 20:02:55.470574 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.470554    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:02:55.505639 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.505613    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:02:55.517873 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.517845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqvb\" (UniqueName: \"kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.517982 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.517913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.619170 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.619132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqvb\" (UniqueName: \"kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.619346 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.619189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.621689 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.621663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.628019 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.627999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqvb\" (UniqueName: \"kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.766869 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.766836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:55.799127 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:55.799099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:02:55.801025 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:02:55.801000    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ed8296_86e0_4a70_bd3c_73a8e12541fd.slice/crio-1c23035b308e693cfc6caa8ad3e3fd30c1cc92306b5b60203a4e9be4432b7e26 WatchSource:0}: Error finding container 1c23035b308e693cfc6caa8ad3e3fd30c1cc92306b5b60203a4e9be4432b7e26: Status 404 returned error can't find the container with id 1c23035b308e693cfc6caa8ad3e3fd30c1cc92306b5b60203a4e9be4432b7e26
Sep 18 20:02:56.040478 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.040456    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:02:56.042625 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:02:56.042592    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a602b58_bdcb_4944_b20d_4e0255cb4f09.slice/crio-5da730051fed1a7b1175613567a2703b6e2131f2c8e7014138f4da2b19bf0889 WatchSource:0}: Error finding container 5da730051fed1a7b1175613567a2703b6e2131f2c8e7014138f4da2b19bf0889: Status 404 returned error can't find the container with id 5da730051fed1a7b1175613567a2703b6e2131f2c8e7014138f4da2b19bf0889
Sep 18 20:02:56.772995 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.772955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" event={"ID":"1a602b58-bdcb-4944-b20d-4e0255cb4f09","Type":"ContainerStarted","Data":"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"}
Sep 18 20:02:56.773446 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.773027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" event={"ID":"1a602b58-bdcb-4944-b20d-4e0255cb4f09","Type":"ContainerStarted","Data":"5da730051fed1a7b1175613567a2703b6e2131f2c8e7014138f4da2b19bf0889"}
Sep 18 20:02:56.773446 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.773057    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:02:56.774822 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.774794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerStarted","Data":"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"}
Sep 18 20:02:56.774941 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.774834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerStarted","Data":"1c23035b308e693cfc6caa8ad3e3fd30c1cc92306b5b60203a4e9be4432b7e26"}
Sep 18 20:02:56.795158 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:56.795112    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podStartSLOduration=1.7950983950000001 podStartE2EDuration="1.795098395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:56.792531496 +0000 UTC m=+2909.029658481" watchObservedRunningTime="2026-09-18 20:02:56.795098395 +0000 UTC m=+2909.032225382"
Sep 18 20:02:58.793954 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:58.793914    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerID="669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658" exitCode=0
Sep 18 20:02:58.794336 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:58.793987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerDied","Data":"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"}
Sep 18 20:02:59.801672 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:59.801631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerStarted","Data":"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"}
Sep 18 20:02:59.827916 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:02:59.827858    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podStartSLOduration=2.735742544 podStartE2EDuration="4.827842776s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.092100232s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:59.825591239 +0000 UTC m=+2912.062718238" watchObservedRunningTime="2026-09-18 20:02:59.827842776 +0000 UTC m=+2912.064969762"
Sep 18 20:03:00.506755 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:00.506689    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:03:05.505872 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:05.505831    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:03:05.507489 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:05.507453    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:03:15.506236 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:15.506186    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:03:25.506472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:25.506427    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:03:27.789425 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:27.789392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:03:35.506895 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:35.506841    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:03:45.507005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:45.506903    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:03:55.506798 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:03:55.506747    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:05.506335 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:05.506293    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:15.506724 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:15.506680    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:25.506745 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:25.506691    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:30.040877 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:30.040845    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:04:30.048126 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:30.048101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:04:30.048499 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:30.048483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:04:30.055900 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:30.055883    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:04:35.506637 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:35.506595    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:45.507123 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:45.507078    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:04:55.516478 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:55.516446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:04:55.524056 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:04:55.524030    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:05:00.317066 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:00.317033    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:05:00.321204 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:00.321178    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:05:07.789621 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:07.789578    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:05:15.339782 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:15.339648    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main" containerID="cri-o://943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3" gracePeriod=45
Sep 18 20:05:15.340396 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:15.340053    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" containerID="cri-o://db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56" gracePeriod=45
Sep 18 20:05:15.517373 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:15.517334    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 18 20:05:16.083357 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.083331    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:05:16.151092 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151026    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151092 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqkgf\" (UniqueName: \"kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151187    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151293 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151242    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache\") pod \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\" (UID: \"b7ed8296-86e0-4a70-bd3c-73a8e12541fd\") "
Sep 18 20:05:16.151611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:16.151670 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.151635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:16.152134 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.152105    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home" (OuterVolumeSpecName: "home") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:16.153582 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.153550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm" (OuterVolumeSpecName: "dshm") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:16.153683 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.153594    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf" (OuterVolumeSpecName: "kube-api-access-jqkgf") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "kube-api-access-jqkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:16.153683 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.153670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:16.171128 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.171102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:16.177961 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.177933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7ed8296-86e0-4a70-bd3c-73a8e12541fd" (UID: "b7ed8296-86e0-4a70-bd3c-73a8e12541fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:16.251976 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.251944    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.251976 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.251969    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.251976 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.251978    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.252164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.251989    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.252164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.251997    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.252164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.252005    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.252164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.252012    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.252164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.252019    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkgf\" (UniqueName: \"kubernetes.io/projected/b7ed8296-86e0-4a70-bd3c-73a8e12541fd-kube-api-access-jqkgf\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:16.488983 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.488951    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerID="db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56" exitCode=0
Sep 18 20:05:16.489378 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.488999    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"
Sep 18 20:05:16.489378 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.489034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerDied","Data":"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"}
Sep 18 20:05:16.489378 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.489090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c" event={"ID":"b7ed8296-86e0-4a70-bd3c-73a8e12541fd","Type":"ContainerDied","Data":"1c23035b308e693cfc6caa8ad3e3fd30c1cc92306b5b60203a4e9be4432b7e26"}
Sep 18 20:05:16.489378 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.489111    2582 scope.go:117] "RemoveContainer" containerID="db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"
Sep 18 20:05:16.498279 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.498262    2582 scope.go:117] "RemoveContainer" containerID="669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"
Sep 18 20:05:16.506164 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.506146    2582 scope.go:117] "RemoveContainer" containerID="db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"
Sep 18 20:05:16.506401 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:05:16.506384    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56\": container with ID starting with db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56 not found: ID does not exist" containerID="db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"
Sep 18 20:05:16.506455 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.506408    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56"} err="failed to get container status \"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56\": rpc error: code = NotFound desc = could not find container \"db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56\": container with ID starting with db05b653774f0ab3354a665b160c4bed1e2f2544898b48df6c6741acc57ccb56 not found: ID does not exist"
Sep 18 20:05:16.506455 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.506425    2582 scope.go:117] "RemoveContainer" containerID="669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"
Sep 18 20:05:16.506654 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:05:16.506633    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658\": container with ID starting with 669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658 not found: ID does not exist" containerID="669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"
Sep 18 20:05:16.506707 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.506664    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658"} err="failed to get container status \"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658\": rpc error: code = NotFound desc = could not find container \"669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658\": container with ID starting with 669165878a6362efb215ee0a8c14992583b7610cb87fe61cdeaa8b93cefed658 not found: ID does not exist"
Sep 18 20:05:16.512457 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.512435    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:05:16.517040 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:16.517020    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-85bc676c96-6jt8c"]
Sep 18 20:05:17.788856 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:17.788823    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:05:17.894605 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:17.894573    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" path="/var/lib/kubelet/pods/b7ed8296-86e0-4a70-bd3c-73a8e12541fd/volumes"
Sep 18 20:05:27.789255 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:27.789211    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:05:27.789694 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:27.789329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:05:37.788983 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:37.788942    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:05:45.587185 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.587158    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:05:45.629874 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.629805    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerID="943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3" exitCode=0
Sep 18 20:05:45.629874 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.629849    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"
Sep 18 20:05:45.630028 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.629894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" event={"ID":"1a602b58-bdcb-4944-b20d-4e0255cb4f09","Type":"ContainerDied","Data":"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"}
Sep 18 20:05:45.630028 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.629950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt" event={"ID":"1a602b58-bdcb-4944-b20d-4e0255cb4f09","Type":"ContainerDied","Data":"5da730051fed1a7b1175613567a2703b6e2131f2c8e7014138f4da2b19bf0889"}
Sep 18 20:05:45.630028 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.629976    2582 scope.go:117] "RemoveContainer" containerID="943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"
Sep 18 20:05:45.640214 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.640196    2582 scope.go:117] "RemoveContainer" containerID="943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"
Sep 18 20:05:45.640519 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:05:45.640502    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3\": container with ID starting with 943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3 not found: ID does not exist" containerID="943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"
Sep 18 20:05:45.640580 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.640527    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3"} err="failed to get container status \"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3\": rpc error: code = NotFound desc = could not find container \"943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3\": container with ID starting with 943c1d678f947775cb43b0bb894f19fb4135f6814005c5a2e2fdc370a25536d3 not found: ID does not exist"
Sep 18 20:05:45.695780 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.695722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs\") pod \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") "
Sep 18 20:05:45.695891 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.695784    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntqvb\" (UniqueName: \"kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb\") pod \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\" (UID: \"1a602b58-bdcb-4944-b20d-4e0255cb4f09\") "
Sep 18 20:05:45.697981 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.697952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb" (OuterVolumeSpecName: "kube-api-access-ntqvb") pod "1a602b58-bdcb-4944-b20d-4e0255cb4f09" (UID: "1a602b58-bdcb-4944-b20d-4e0255cb4f09"). InnerVolumeSpecName "kube-api-access-ntqvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:45.698065 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.697990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a602b58-bdcb-4944-b20d-4e0255cb4f09" (UID: "1a602b58-bdcb-4944-b20d-4e0255cb4f09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:45.796519 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.796481    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a602b58-bdcb-4944-b20d-4e0255cb4f09-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:45.796519 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.796515    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqvb\" (UniqueName: \"kubernetes.io/projected/1a602b58-bdcb-4944-b20d-4e0255cb4f09-kube-api-access-ntqvb\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:05:45.947839 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.947803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:05:45.951750 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:45.951712    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-hhdmt"]
Sep 18 20:05:47.894129 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:05:47.894100    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" path="/var/lib/kubelet/pods/1a602b58-bdcb-4944-b20d-4e0255cb4f09/volumes"
Sep 18 20:06:02.870223 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870187    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870470    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870481    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870499    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870504    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870511    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="storage-initializer"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870517    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="storage-initializer"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870562    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ed8296-86e0-4a70-bd3c-73a8e12541fd" containerName="main"
Sep 18 20:06:02.870635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.870570    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a602b58-bdcb-4944-b20d-4e0255cb4f09" containerName="main"
Sep 18 20:06:02.873541 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.873515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.876297 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.876265    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-k2cdl\""
Sep 18 20:06:02.876464 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.876428    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 20:06:02.876594 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.876428    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-jvfl6\""
Sep 18 20:06:02.876715 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.876700    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 20:06:02.877045 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.877027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:06:02.877137 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.877027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 20:06:02.880672 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.880650    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 20:06:02.900276 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.900252    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:06:02.908541 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.908519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:06:02.912116 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.912096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:02.915948 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.915922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916064 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.915983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916064 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916187 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfbq\" (UniqueName: \"kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916187 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916187 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916345 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.916345 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.916236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:02.930916 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:02.930885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:06:03.017380 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.017380 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ddq\" (UniqueName: \"kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.017632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfbq\" (UniqueName: \"kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.017981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.018008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018451 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.018164    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018495 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.018471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.018590 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.018574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.019973 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.019958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.020400 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.020379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.034053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.034030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfbq\" (UniqueName: \"kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.119151 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ddq\" (UniqueName: \"kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.119502 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.120003 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.120003 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.120003 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.120003 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.119894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.120290 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.120247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.121831 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.121814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.122097 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.122080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.129398 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.129376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ddq\" (UniqueName: \"kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.183526 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.183493    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:03.221998 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.221953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:03.474518 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.474490    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:06:03.475206 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:06:03.475179    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c8448d_90e9_4ad0_9f1d_c190b9133511.slice/crio-9ec6f8498698318a0d7e5e023cf090c8557946da13f2e4069ecffec788358952 WatchSource:0}: Error finding container 9ec6f8498698318a0d7e5e023cf090c8557946da13f2e4069ecffec788358952: Status 404 returned error can't find the container with id 9ec6f8498698318a0d7e5e023cf090c8557946da13f2e4069ecffec788358952
Sep 18 20:06:03.477192 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.477170    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:06:03.533199 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.533173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:06:03.533314 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:06:03.533291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab5b820_182b_4c89_bdcd_5b8b6c5c671d.slice/crio-180f0b2bf2f502b35bb5f9812a724f1c6b9c2220013618781fb883bb51b74a55 WatchSource:0}: Error finding container 180f0b2bf2f502b35bb5f9812a724f1c6b9c2220013618781fb883bb51b74a55: Status 404 returned error can't find the container with id 180f0b2bf2f502b35bb5f9812a724f1c6b9c2220013618781fb883bb51b74a55
Sep 18 20:06:03.716332 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.716290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerStarted","Data":"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"}
Sep 18 20:06:03.716528 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.716352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerStarted","Data":"180f0b2bf2f502b35bb5f9812a724f1c6b9c2220013618781fb883bb51b74a55"}
Sep 18 20:06:03.717632 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:03.717607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerStarted","Data":"9ec6f8498698318a0d7e5e023cf090c8557946da13f2e4069ecffec788358952"}
Sep 18 20:06:05.730008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:05.729969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerStarted","Data":"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"}
Sep 18 20:06:05.730465 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:05.730105    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:06.736752 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:06.736694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerStarted","Data":"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"}
Sep 18 20:06:06.738750 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:06.738695    2582 generic.go:356] "Generic (PLEG): container finished" podID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerID="10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee" exitCode=0
Sep 18 20:06:06.738936 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:06.738775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerDied","Data":"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"}
Sep 18 20:06:07.747533 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:07.747492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerStarted","Data":"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"}
Sep 18 20:06:07.781596 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:07.781536    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podStartSLOduration=3.381197611 podStartE2EDuration="5.781510408s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.400312797s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:07.779259671 +0000 UTC m=+3100.016386658" watchObservedRunningTime="2026-09-18 20:06:07.781510408 +0000 UTC m=+3100.018637395"
Sep 18 20:06:08.754572 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:08.754533    2582 generic.go:356] "Generic (PLEG): container finished" podID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerID="e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d" exitCode=0
Sep 18 20:06:08.755093 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:08.754603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerDied","Data":"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"}
Sep 18 20:06:09.762784 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:09.762742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerStarted","Data":"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"}
Sep 18 20:06:09.795346 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:09.795278    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podStartSLOduration=4.133943775 podStartE2EDuration="7.795258506s" totalImagesPullingTime="1.442192457s" totalInitContainerRuntime="2.219122274s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:09.792307155 +0000 UTC m=+3102.029434142" watchObservedRunningTime="2026-09-18 20:06:09.795258506 +0000 UTC m=+3102.032385494"
Sep 18 20:06:13.184161 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.184120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:13.184762 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.184191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:13.185849 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.185819    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:06:13.222128 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.222096    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:13.222278 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.222152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:06:13.223611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:13.223582    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:06:23.184983 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:23.184925    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:06:23.202816 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:23.202790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:06:23.222532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:23.222491    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:06:33.185000 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:33.184950    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:06:33.222307 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:33.222262    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:06:43.184287 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:43.184240    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:06:43.222423 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:43.222375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:06:53.184348 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:53.184298    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:06:53.222495 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:06:53.222456    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:03.184547 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:03.184495    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:03.222262 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:03.222224    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:13.184663 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:13.184616    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:13.222319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:13.222250    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:23.184234 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:23.184180    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:23.222340 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:23.222295    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:33.184127 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:33.184073    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:33.222388 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:33.222350    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:43.184380 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:43.184329    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:43.222310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:43.222269    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:07:53.184185 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:53.184142    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:07:53.222260 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:07:53.222208    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:08:03.184596 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:03.184548    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:08:03.222345 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:03.222300    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:08:13.184064 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:13.183962    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:08:13.222372 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:13.222335    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:08:23.184366 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:23.184319    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:08:23.222385 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:23.222338    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:08:33.184887 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:33.184837    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 18 20:08:33.222398 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:33.222354    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 18 20:08:43.193871 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:43.193834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:08:43.205832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:43.205811    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:08:43.232400 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:43.232365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:08:43.240107 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:43.240084    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:08:52.586457 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:52.586418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:08:52.588534 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:08:52.588504    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:09:07.609944 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:07.609880    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main" containerID="cri-o://aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd" gracePeriod=45
Sep 18 20:09:07.610614 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:07.610453    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main" containerID="cri-o://81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da" gracePeriod=45
Sep 18 20:09:08.427839 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.427792    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="llm-d-routing-sidecar" containerID="cri-o://5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1" gracePeriod=45
Sep 18 20:09:08.554795 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.554766    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:09:08.668526 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.668497    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:09:08.707779 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707722    2582 generic.go:356] "Generic (PLEG): container finished" podID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerID="81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da" exitCode=0
Sep 18 20:09:08.707779 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707777    2582 generic.go:356] "Generic (PLEG): container finished" podID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerID="5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1" exitCode=0
Sep 18 20:09:08.707974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerDied","Data":"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"}
Sep 18 20:09:08.707974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707821    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"
Sep 18 20:09:08.707974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerDied","Data":"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"}
Sep 18 20:09:08.707974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn" event={"ID":"68c8448d-90e9-4ad0-9f1d-c190b9133511","Type":"ContainerDied","Data":"9ec6f8498698318a0d7e5e023cf090c8557946da13f2e4069ecffec788358952"}
Sep 18 20:09:08.707974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.707902    2582 scope.go:117] "RemoveContainer" containerID="81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"
Sep 18 20:09:08.708822 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.708799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.708990 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.708835    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.708990 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.708979    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709257 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709257 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709257 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709441 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72ddq\" (UniqueName: \"kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709441 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs\") pod \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\" (UID: \"dab5b820-182b-4c89-bdcd-5b8b6c5c671d\") "
Sep 18 20:09:08.709848 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home" (OuterVolumeSpecName: "home") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.709982 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.709958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:08.710223 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.710202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.710527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.710339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerDied","Data":"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"}
Sep 18 20:09:08.710527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.710318    2582 generic.go:356] "Generic (PLEG): container finished" podID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerID="aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd" exitCode=0
Sep 18 20:09:08.710527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.710428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4" event={"ID":"dab5b820-182b-4c89-bdcd-5b8b6c5c671d","Type":"ContainerDied","Data":"180f0b2bf2f502b35bb5f9812a724f1c6b9c2220013618781fb883bb51b74a55"}
Sep 18 20:09:08.710527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.710517    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"
Sep 18 20:09:08.712429 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.712397    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm" (OuterVolumeSpecName: "dshm") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.712953 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.712896    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:08.712953 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.712939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq" (OuterVolumeSpecName: "kube-api-access-72ddq") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "kube-api-access-72ddq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:08.719299 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.719277    2582 scope.go:117] "RemoveContainer" containerID="e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"
Sep 18 20:09:08.724002 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.723974    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.727790 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.727768    2582 scope.go:117] "RemoveContainer" containerID="5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"
Sep 18 20:09:08.736686 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.736656    2582 scope.go:117] "RemoveContainer" containerID="81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"
Sep 18 20:09:08.736991 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:09:08.736969    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da\": container with ID starting with 81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da not found: ID does not exist" containerID="81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"
Sep 18 20:09:08.737051 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737000    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"} err="failed to get container status \"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da\": rpc error: code = NotFound desc = could not find container \"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da\": container with ID starting with 81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da not found: ID does not exist"
Sep 18 20:09:08.737051 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737018    2582 scope.go:117] "RemoveContainer" containerID="e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"
Sep 18 20:09:08.737263 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:09:08.737242    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d\": container with ID starting with e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d not found: ID does not exist" containerID="e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"
Sep 18 20:09:08.737311 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737271    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"} err="failed to get container status \"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d\": rpc error: code = NotFound desc = could not find container \"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d\": container with ID starting with e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d not found: ID does not exist"
Sep 18 20:09:08.737311 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737287    2582 scope.go:117] "RemoveContainer" containerID="5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"
Sep 18 20:09:08.737490 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:09:08.737472    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1\": container with ID starting with 5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1 not found: ID does not exist" containerID="5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"
Sep 18 20:09:08.737537 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737495    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"} err="failed to get container status \"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1\": rpc error: code = NotFound desc = could not find container \"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1\": container with ID starting with 5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1 not found: ID does not exist"
Sep 18 20:09:08.737537 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737508    2582 scope.go:117] "RemoveContainer" containerID="81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"
Sep 18 20:09:08.737773 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737754    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da"} err="failed to get container status \"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da\": rpc error: code = NotFound desc = could not find container \"81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da\": container with ID starting with 81177b7963530a57143851338322ecfa86b3933357e68cf5b904b8579b6114da not found: ID does not exist"
Sep 18 20:09:08.737835 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737775    2582 scope.go:117] "RemoveContainer" containerID="e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"
Sep 18 20:09:08.738014 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.737994    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d"} err="failed to get container status \"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d\": rpc error: code = NotFound desc = could not find container \"e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d\": container with ID starting with e3a883e159be3e6fb5885d25acc209ccf6a22a838aeb6c0846cc5f62254dbe8d not found: ID does not exist"
Sep 18 20:09:08.738064 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.738016    2582 scope.go:117] "RemoveContainer" containerID="5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"
Sep 18 20:09:08.738190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.738169    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dab5b820-182b-4c89-bdcd-5b8b6c5c671d" (UID: "dab5b820-182b-4c89-bdcd-5b8b6c5c671d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.738236 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.738185    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1"} err="failed to get container status \"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1\": rpc error: code = NotFound desc = could not find container \"5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1\": container with ID starting with 5591adbea0e60b5408508d786bcb9558a5e5639bf78e467b16b242c1939303b1 not found: ID does not exist"
Sep 18 20:09:08.738236 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.738198    2582 scope.go:117] "RemoveContainer" containerID="aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"
Sep 18 20:09:08.745780 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.745761    2582 scope.go:117] "RemoveContainer" containerID="10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"
Sep 18 20:09:08.753420 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.753403    2582 scope.go:117] "RemoveContainer" containerID="aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"
Sep 18 20:09:08.753680 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:09:08.753661    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd\": container with ID starting with aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd not found: ID does not exist" containerID="aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"
Sep 18 20:09:08.753765 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.753683    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd"} err="failed to get container status \"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd\": rpc error: code = NotFound desc = could not find container \"aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd\": container with ID starting with aa92125bf95e137df888061edc5e9814be6d641074c4257aa26b06c67a228ddd not found: ID does not exist"
Sep 18 20:09:08.753765 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.753699    2582 scope.go:117] "RemoveContainer" containerID="10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"
Sep 18 20:09:08.753969 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:09:08.753954    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee\": container with ID starting with 10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee not found: ID does not exist" containerID="10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"
Sep 18 20:09:08.754019 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.753973    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee"} err="failed to get container status \"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee\": rpc error: code = NotFound desc = could not find container \"10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee\": container with ID starting with 10751fca5f6f45e92e72f70aedab73d8d8d4234c1e5dbdd62c71ba95aff7c0ee not found: ID does not exist"
Sep 18 20:09:08.810310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810264    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810331    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810469 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810688 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810688 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kfbq\" (UniqueName: \"kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810688 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir\") pod \"68c8448d-90e9-4ad0-9f1d-c190b9133511\" (UID: \"68c8448d-90e9-4ad0-9f1d-c190b9133511\") "
Sep 18 20:09:08.810878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810804    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache" (OuterVolumeSpecName: "model-cache") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.810951 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:08.810951 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810923    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.810951 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810948    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810963    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810974    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.810986    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.811002    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.811021    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.811038    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ddq\" (UniqueName: \"kubernetes.io/projected/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-kube-api-access-72ddq\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.811049    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab5b820-182b-4c89-bdcd-5b8b6c5c671d-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.811427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.811346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home" (OuterVolumeSpecName: "home") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.812826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.812799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm" (OuterVolumeSpecName: "dshm") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.813053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.813034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:08.813154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.813131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq" (OuterVolumeSpecName: "kube-api-access-6kfbq") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "kube-api-access-6kfbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:08.824791 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.824767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.834148 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.834127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68c8448d-90e9-4ad0-9f1d-c190b9133511" (UID: "68c8448d-90e9-4ad0-9f1d-c190b9133511"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912225    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912258    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912268    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68c8448d-90e9-4ad0-9f1d-c190b9133511-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912277    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912285    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c8448d-90e9-4ad0-9f1d-c190b9133511-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912292    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfbq\" (UniqueName: \"kubernetes.io/projected/68c8448d-90e9-4ad0-9f1d-c190b9133511-kube-api-access-6kfbq\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:08.912310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:08.912299    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c8448d-90e9-4ad0-9f1d-c190b9133511-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:09:09.034821 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.034791    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:09:09.040745 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.040703    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-68d6bbddd8jgbgn"]
Sep 18 20:09:09.053980 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.053954    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:09:09.059034 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.059006    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58zm6n4"]
Sep 18 20:09:09.895485 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.895450    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" path="/var/lib/kubelet/pods/68c8448d-90e9-4ad0-9f1d-c190b9133511/volumes"
Sep 18 20:09:09.895982 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:09.895967    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" path="/var/lib/kubelet/pods/dab5b820-182b-4c89-bdcd-5b8b6c5c671d/volumes"
Sep 18 20:09:25.581064 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581032    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581302    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581314    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581325    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="llm-d-routing-sidecar"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581330    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="llm-d-routing-sidecar"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581343    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="storage-initializer"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581348    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="storage-initializer"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581354    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581359    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581372    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="storage-initializer"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581377    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="storage-initializer"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581425    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="llm-d-routing-sidecar"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581432    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c8448d-90e9-4ad0-9f1d-c190b9133511" containerName="main"
Sep 18 20:09:25.581532 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.581441    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab5b820-182b-4c89-bdcd-5b8b6c5c671d" containerName="main"
Sep 18 20:09:25.586699 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.586678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.604210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.604184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-wq2hl\""
Sep 18 20:09:25.620900 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.620878    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:09:25.621059 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.621041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 18 20:09:25.621146 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.621052    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 18 20:09:25.621313 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.621297    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 18 20:09:25.628510 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.628487    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 18 20:09:25.634058 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634155 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634155 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634155 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634155 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634302 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634302 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.634367 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.634314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77hg\" (UniqueName: \"kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735279 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735394 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735394 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735394 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735394 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77hg\" (UniqueName: \"kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735394 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735669 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735669 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735798 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735890 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.735961 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.736012 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.735954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.736246 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.736150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.737706 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.737684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.738143 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.738123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.802008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.801977    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:09:25.813542 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.813522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77hg\" (UniqueName: \"kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg\") pod \"lora-multiple-adapters-test-kserve-55f96495bc-fpgt2\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:25.896138 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:25.896059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:26.220479 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:26.220454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:09:26.223599 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:09:26.223562    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fcb8b9_fe40_420f_a761_b7de44f69722.slice/crio-966e7fc48eea29e952836e4878be403fd1af76164ce6f2c4a681006157463f32 WatchSource:0}: Error finding container 966e7fc48eea29e952836e4878be403fd1af76164ce6f2c4a681006157463f32: Status 404 returned error can't find the container with id 966e7fc48eea29e952836e4878be403fd1af76164ce6f2c4a681006157463f32
Sep 18 20:09:26.799190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:26.799157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerStarted","Data":"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"}
Sep 18 20:09:26.799569 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:26.799208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerStarted","Data":"966e7fc48eea29e952836e4878be403fd1af76164ce6f2c4a681006157463f32"}
Sep 18 20:09:29.814062 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:29.814033    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerID="f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229" exitCode=0
Sep 18 20:09:29.814435 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:29.814068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerDied","Data":"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"}
Sep 18 20:09:30.084005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.083913    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:09:30.090067 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.090037    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:09:30.091692 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.091671    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:09:30.097319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.097304    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:09:30.821600 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.821562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerStarted","Data":"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"}
Sep 18 20:09:30.858592 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.858541    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podStartSLOduration=2.940305792 podStartE2EDuration="5.858526302s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.91822051s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:30.854187334 +0000 UTC m=+3303.091314323" watchObservedRunningTime="2026-09-18 20:09:30.858526302 +0000 UTC m=+3303.095653287"
Sep 18 20:09:30.896845 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:30.896802    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:35.896247 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:35.896215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:09:35.897855 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:35.897818    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:09:45.896509 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:45.896415    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:09:55.897291 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:09:55.897246    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:10:02.373609 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.373575    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:10:02.375926 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.375908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.381409 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.381376    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:10:02.381534 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.381473    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 20:10:02.381842 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.381819    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xz8hh\""
Sep 18 20:10:02.381937 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.381846    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:10:02.382003 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.381965    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:10:02.382641 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.382481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:10:02.425065 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.425034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:10:02.447098 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447098 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447301 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447301 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447301 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67mn\" (UniqueName: \"kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.447427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.447373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548321 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67mn\" (UniqueName: \"kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548321 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.548907 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.549039 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.548937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.549113 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.549036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.549113 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.549110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.549321 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.549301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.550984 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.550965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.551717 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.551697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.564338 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.564316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67mn\" (UniqueName: \"kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn\") pod \"custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.686472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.686389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:02.985695 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:02.985664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:10:02.988778 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:10:02.988714    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d0b5f_e7b7_4d2b_8422_feae0fbf610a.slice/crio-357c32977284ad95b265325fcf218b5a9967750f0fa6a901b66eb7c8d0238432 WatchSource:0}: Error finding container 357c32977284ad95b265325fcf218b5a9967750f0fa6a901b66eb7c8d0238432: Status 404 returned error can't find the container with id 357c32977284ad95b265325fcf218b5a9967750f0fa6a901b66eb7c8d0238432
Sep 18 20:10:03.994196 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:03.994154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerStarted","Data":"46d45281cd249124ed25e273aed14d74279e13193b305a288ffbffee14a1873e"}
Sep 18 20:10:03.994678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:03.994207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerStarted","Data":"357c32977284ad95b265325fcf218b5a9967750f0fa6a901b66eb7c8d0238432"}
Sep 18 20:10:05.896698 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:05.896657    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:10:06.007677 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:06.007647    2582 generic.go:356] "Generic (PLEG): container finished" podID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerID="46d45281cd249124ed25e273aed14d74279e13193b305a288ffbffee14a1873e" exitCode=0
Sep 18 20:10:06.007892 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:06.007741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerDied","Data":"46d45281cd249124ed25e273aed14d74279e13193b305a288ffbffee14a1873e"}
Sep 18 20:10:07.015504 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:07.015462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerStarted","Data":"00e269f17c652d263db2479b6a371c6f3e13d8e55936dd11c62c0cb34481f257"}
Sep 18 20:10:07.083800 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:07.083707    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podStartSLOduration=2.805584424 podStartE2EDuration="5.083684287s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.278099863s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:07.079228153 +0000 UTC m=+3339.316355139" watchObservedRunningTime="2026-09-18 20:10:07.083684287 +0000 UTC m=+3339.320811274"
Sep 18 20:10:12.686964 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:12.686916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:12.687363 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:12.687010    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:10:12.688683 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:12.688651    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:10:15.897069 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:15.897022    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:10:22.687435 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:22.687386    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:10:25.897218 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:25.897173    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:10:32.687084 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:32.687037    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:10:35.896529 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:35.896480    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:10:42.687562 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:42.687512    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:10:45.906999 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:45.906969    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:10:45.914702 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:45.914670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:10:52.687592 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:10:52.687536    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:02.687883 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:02.687823    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:12.687013 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:12.686900    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:12.960758 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:12.960620    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55f96495bc-fpgt2_e3fcb8b9-fe40-420f-a761-b7de44f69722/storage-initializer/0.log"
Sep 18 20:11:12.974635 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:12.974607    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55f96495bc-fpgt2_e3fcb8b9-fe40-420f-a761-b7de44f69722/main/0.log"
Sep 18 20:11:14.353854 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:14.353820    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:11:22.687797 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:22.687751    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:29.376748 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:29.376678    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" containerID="cri-o://1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27" gracePeriod=45
Sep 18 20:11:29.908113 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:29.908072    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 20:11:30.328535 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.328509    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:11:30.415850 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.415821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.415860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.415888    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.415912    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.415974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m77hg\" (UniqueName: \"kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416326 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416147    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:30.416659 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home" (OuterVolumeSpecName: "home") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:30.416659 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416659 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert\") pod \"e3fcb8b9-fe40-420f-a761-b7de44f69722\" (UID: \"e3fcb8b9-fe40-420f-a761-b7de44f69722\") "
Sep 18 20:11:30.416901 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416881    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.416994 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.416904    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.417439 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.417230    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:11:30.419483 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.419455    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:30.419601 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.419480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg" (OuterVolumeSpecName: "kube-api-access-m77hg") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "kube-api-access-m77hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:30.419601 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.419546    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm" (OuterVolumeSpecName: "dshm") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:30.428413 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.428381    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:30.446229 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.446160    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3fcb8b9-fe40-420f-a761-b7de44f69722" (UID: "e3fcb8b9-fe40-420f-a761-b7de44f69722"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:30.481842 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.481803    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerID="1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27" exitCode=0
Sep 18 20:11:30.481975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.481850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"
Sep 18 20:11:30.481975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.481886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerDied","Data":"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"}
Sep 18 20:11:30.481975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.481951    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2" event={"ID":"e3fcb8b9-fe40-420f-a761-b7de44f69722","Type":"ContainerDied","Data":"966e7fc48eea29e952836e4878be403fd1af76164ce6f2c4a681006157463f32"}
Sep 18 20:11:30.482090 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.481984    2582 scope.go:117] "RemoveContainer" containerID="1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"
Sep 18 20:11:30.491910 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.491890    2582 scope.go:117] "RemoveContainer" containerID="f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"
Sep 18 20:11:30.500413 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.500395    2582 scope.go:117] "RemoveContainer" containerID="1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"
Sep 18 20:11:30.500766 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:11:30.500722    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27\": container with ID starting with 1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27 not found: ID does not exist" containerID="1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"
Sep 18 20:11:30.500838 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.500775    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27"} err="failed to get container status \"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27\": rpc error: code = NotFound desc = could not find container \"1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27\": container with ID starting with 1cad580482804f79f1b816cce524d7b9eb5affef45762d30ed020fcd7ec76c27 not found: ID does not exist"
Sep 18 20:11:30.500838 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.500801    2582 scope.go:117] "RemoveContainer" containerID="f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"
Sep 18 20:11:30.501067 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:11:30.501045    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229\": container with ID starting with f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229 not found: ID does not exist" containerID="f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"
Sep 18 20:11:30.501127 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.501078    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229"} err="failed to get container status \"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229\": rpc error: code = NotFound desc = could not find container \"f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229\": container with ID starting with f124c8923b6614dca2e341371f63cb0fda41a09b39fd849468133645d7b9d229 not found: ID does not exist"
Sep 18 20:11:30.512555 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.511967    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:11:30.517151 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517130    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f96495bc-fpgt2"]
Sep 18 20:11:30.517480 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517456    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.517524 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517513    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fcb8b9-fe40-420f-a761-b7de44f69722-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.517560 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517532    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.517560 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517549    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77hg\" (UniqueName: \"kubernetes.io/projected/e3fcb8b9-fe40-420f-a761-b7de44f69722-kube-api-access-m77hg\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.517627 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517564    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3fcb8b9-fe40-420f-a761-b7de44f69722-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:30.517627 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:30.517579    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3fcb8b9-fe40-420f-a761-b7de44f69722-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:31.895355 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:31.895317    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" path="/var/lib/kubelet/pods/e3fcb8b9-fe40-420f-a761-b7de44f69722/volumes"
Sep 18 20:11:32.687258 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:32.687217    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:37.978537 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978500    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"]
Sep 18 20:11:37.979031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978802    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main"
Sep 18 20:11:37.979031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978813    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main"
Sep 18 20:11:37.979031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978823    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="storage-initializer"
Sep 18 20:11:37.979031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978828    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="storage-initializer"
Sep 18 20:11:37.979031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.978876    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3fcb8b9-fe40-420f-a761-b7de44f69722" containerName="main"
Sep 18 20:11:37.983495 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.983475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:37.989693 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.989370    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 20:11:37.990208 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.990186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 20:11:37.990208 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.990193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 20:11:37.990380 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:37.990193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-tmrfj\""
Sep 18 20:11:38.000528 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:38.000500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"]
Sep 18 20:11:38.076618 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:38.076587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtz2\" (UniqueName: \"kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:38.177620 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:38.177580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtz2\" (UniqueName: \"kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:38.189133 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:38.189111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtz2\" (UniqueName: \"kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:42.687170 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:42.687121    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:11:43.722850 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:43.722800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:43.733136 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:43.733106    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:43.823505 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:43.823467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:43.823708 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:43.823620    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:43.830086 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:43.830062    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") DevicePath \"csi-559c9459636d88d22043d689b5f71bf416c602502cae5a7763f62b42a7515d0f\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:44.053010 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:44.052973    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9ab8b061cad53f4ee0be5078fb529b069c364e88ab2770f2ad5db58aa593302a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:44.072694 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:44.072661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"e2e-pvc-lora-download-qgw2b\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:44.292633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:44.292588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:44.597423 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:44.597324    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"]
Sep 18 20:11:44.601863 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:11:44.601832    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa03d1ee_8233_420e_a15a_88c9dc4ef029.slice/crio-f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3 WatchSource:0}: Error finding container f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3: Status 404 returned error can't find the container with id f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3
Sep 18 20:11:45.578198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:45.578151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b" event={"ID":"aa03d1ee-8233-420e-a15a-88c9dc4ef029","Type":"ContainerStarted","Data":"78a8751541bf9f9e9ea22e39aa9392f709a5ea69276db8d80c7f3319cb53226f"}
Sep 18 20:11:45.578198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:45.578206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b" event={"ID":"aa03d1ee-8233-420e-a15a-88c9dc4ef029","Type":"ContainerStarted","Data":"f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3"}
Sep 18 20:11:45.619090 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:45.619017    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b" podStartSLOduration=11.618997001 podStartE2EDuration="11.618997001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:45.617423661 +0000 UTC m=+3437.854550649" watchObservedRunningTime="2026-09-18 20:11:45.618997001 +0000 UTC m=+3437.856123988"
Sep 18 20:11:47.589670 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:47.589633    2582 generic.go:356] "Generic (PLEG): container finished" podID="aa03d1ee-8233-420e-a15a-88c9dc4ef029" containerID="78a8751541bf9f9e9ea22e39aa9392f709a5ea69276db8d80c7f3319cb53226f" exitCode=0
Sep 18 20:11:47.590205 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:47.589710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b" event={"ID":"aa03d1ee-8233-420e-a15a-88c9dc4ef029","Type":"ContainerDied","Data":"78a8751541bf9f9e9ea22e39aa9392f709a5ea69276db8d80c7f3319cb53226f"}
Sep 18 20:11:48.728130 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.728103    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:48.766602 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.766566    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") "
Sep 18 20:11:48.766813 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.766658    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvtz2\" (UniqueName: \"kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2\") pod \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\" (UID: \"aa03d1ee-8233-420e-a15a-88c9dc4ef029\") "
Sep 18 20:11:48.769108 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.769078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2" (OuterVolumeSpecName: "kube-api-access-fvtz2") pod "aa03d1ee-8233-420e-a15a-88c9dc4ef029" (UID: "aa03d1ee-8233-420e-a15a-88c9dc4ef029"). InnerVolumeSpecName "kube-api-access-fvtz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:48.771023 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.770998    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492" (OuterVolumeSpecName: "model-storage") pod "aa03d1ee-8233-420e-a15a-88c9dc4ef029" (UID: "aa03d1ee-8233-420e-a15a-88c9dc4ef029"). InnerVolumeSpecName "pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 20:11:48.867995 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.867911    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtz2\" (UniqueName: \"kubernetes.io/projected/aa03d1ee-8233-420e-a15a-88c9dc4ef029-kube-api-access-fvtz2\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:11:48.867995 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:48.867977    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") on node \"ip-10-0-143-75.ec2.internal\" "
Sep 18 20:11:49.601856 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:49.601827    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"
Sep 18 20:11:49.602024 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:49.601884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b" event={"ID":"aa03d1ee-8233-420e-a15a-88c9dc4ef029","Type":"ContainerDied","Data":"f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3"}
Sep 18 20:11:49.602024 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:49.601921    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4dd65f0dd79aad2a58cd9264ed58c806dc4b49aa9b5cf277e86159a09887bf3"
Sep 18 20:11:49.839031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:49.839006    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492") on node "ip-10-0-143-75.ec2.internal"
Sep 18 20:11:49.876466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:49.876372    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"csi-559c9459636d88d22043d689b5f71bf416c602502cae5a7763f62b42a7515d0f\""
Sep 18 20:11:52.687406 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:11:52.687353    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 18 20:12:02.181722 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.181687    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:12:02.182095 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.182009    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa03d1ee-8233-420e-a15a-88c9dc4ef029" containerName="storage-initializer"
Sep 18 20:12:02.182095 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.182020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa03d1ee-8233-420e-a15a-88c9dc4ef029" containerName="storage-initializer"
Sep 18 20:12:02.182095 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.182079    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa03d1ee-8233-420e-a15a-88c9dc4ef029" containerName="storage-initializer"
Sep 18 20:12:02.184887 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.184869    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.187591 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.187563    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 20:12:02.187753 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.187564    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-tmrfj\""
Sep 18 20:12:02.187753 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.187626    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 20:12:02.187964 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.187948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 20:12:02.188371 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.188350    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:12:02.188491 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.188377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 18 20:12:02.202526 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.202496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:12:02.280464 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280625 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280625 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280625 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280625 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.280885 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.280784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582sj\" (UniqueName: \"kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.381624 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.381831 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582sj\" (UniqueName: \"kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.381831 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.381928 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.381928 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.381960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382219 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382307 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382385 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382424 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.382684 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.382659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.384429 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.384402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.384525 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.384435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.393264 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.393236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-582sj\" (UniqueName: \"kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:02.698047 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.698007    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:12:02.707059 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:02.707026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:12:04.603306 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.603264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.610551 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.610519    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.704243 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.704199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.704404 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.704337    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.707466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.707445    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") DevicePath \"csi-559c9459636d88d22043d689b5f71bf416c602502cae5a7763f62b42a7515d0f\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.798253 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.798218    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9ab8b061cad53f4ee0be5078fb529b069c364e88ab2770f2ad5db58aa593302a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.817113 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.817084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"lora-pvc-single-adapter-test-kserve-55c565b586-zghnx\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:04.895102 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:04.895006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:05.183714 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:05.183688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:12:05.185496 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:12:05.185468    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fa2b40_bd32_40a0_9006_b53544cee4b1.slice/crio-99e1f6d08ebfe9fe1733ca306c7d9b8d729fb312f706bf4337fd42ce582781a9 WatchSource:0}: Error finding container 99e1f6d08ebfe9fe1733ca306c7d9b8d729fb312f706bf4337fd42ce582781a9: Status 404 returned error can't find the container with id 99e1f6d08ebfe9fe1733ca306c7d9b8d729fb312f706bf4337fd42ce582781a9
Sep 18 20:12:05.691377 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:05.691333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerStarted","Data":"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"}
Sep 18 20:12:05.691796 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:05.691439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerStarted","Data":"99e1f6d08ebfe9fe1733ca306c7d9b8d729fb312f706bf4337fd42ce582781a9"}
Sep 18 20:12:07.704070 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:07.703979    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerID="426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd" exitCode=0
Sep 18 20:12:07.704070 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:07.704053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerDied","Data":"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"}
Sep 18 20:12:08.711973 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:08.711926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerStarted","Data":"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"}
Sep 18 20:12:08.743878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:08.743829    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podStartSLOduration=4.674022638 podStartE2EDuration="6.743814207s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.069791569s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:08.741253901 +0000 UTC m=+3460.978380888" watchObservedRunningTime="2026-09-18 20:12:08.743814207 +0000 UTC m=+3460.980941193"
Sep 18 20:12:08.895661 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:08.895631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:14.895411 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:14.895368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:12:14.897249 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:14.897220    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:12:21.137339 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:21.137301    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:12:24.896043 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:24.895999    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:12:34.896082 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:34.896034    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:12:36.160245 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.160161    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main" containerID="cri-o://00e269f17c652d263db2479b6a371c6f3e13d8e55936dd11c62c0cb34481f257" gracePeriod=45
Sep 18 20:12:36.865451 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.865420    2582 generic.go:356] "Generic (PLEG): container finished" podID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerID="00e269f17c652d263db2479b6a371c6f3e13d8e55936dd11c62c0cb34481f257" exitCode=0
Sep 18 20:12:36.865595 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.865467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerDied","Data":"00e269f17c652d263db2479b6a371c6f3e13d8e55936dd11c62c0cb34481f257"}
Sep 18 20:12:36.912985 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.912954    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:12:36.979527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q67mn\" (UniqueName: \"kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979669    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979973 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979704    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.979973 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.979760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert\") pod \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\" (UID: \"745d0b5f-e7b7-4d2b-8422-feae0fbf610a\") "
Sep 18 20:12:36.980168 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.980107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache" (OuterVolumeSpecName: "model-cache") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:36.980168 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.980150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home" (OuterVolumeSpecName: "home") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:36.980367 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.980346    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:36.980429 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.980366    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:36.980684 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.980658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:36.982404 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.982376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm" (OuterVolumeSpecName: "dshm") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:36.982512 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.982376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn" (OuterVolumeSpecName: "kube-api-access-q67mn") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "kube-api-access-q67mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:36.982678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.982654    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:36.993286 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:36.993261    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.002825 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.002710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "745d0b5f-e7b7-4d2b-8422-feae0fbf610a" (UID: "745d0b5f-e7b7-4d2b-8422-feae0fbf610a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:37.080791 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080761    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.080791 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080789    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.080975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080799    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.080975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080806    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.080975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080815    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67mn\" (UniqueName: \"kubernetes.io/projected/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kube-api-access-q67mn\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.080975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.080824    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/745d0b5f-e7b7-4d2b-8422-feae0fbf610a-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:12:37.872254 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.872226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"
Sep 18 20:12:37.872254 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.872232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr" event={"ID":"745d0b5f-e7b7-4d2b-8422-feae0fbf610a","Type":"ContainerDied","Data":"357c32977284ad95b265325fcf218b5a9967750f0fa6a901b66eb7c8d0238432"}
Sep 18 20:12:37.872783 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.872291    2582 scope.go:117] "RemoveContainer" containerID="00e269f17c652d263db2479b6a371c6f3e13d8e55936dd11c62c0cb34481f257"
Sep 18 20:12:37.882081 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.882057    2582 scope.go:117] "RemoveContainer" containerID="46d45281cd249124ed25e273aed14d74279e13193b305a288ffbffee14a1873e"
Sep 18 20:12:37.916536 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.916507    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:12:37.918357 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:37.918337    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-659f4c7cc9-qpbrr"]
Sep 18 20:12:39.895505 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:39.895428    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" path="/var/lib/kubelet/pods/745d0b5f-e7b7-4d2b-8422-feae0fbf610a/volumes"
Sep 18 20:12:44.896431 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:44.896375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:12:54.896425 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:12:54.896382    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:13:04.895840 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:04.895791    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:13:14.895803 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:14.895758    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:13:24.905884 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.905853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:13:24.913539 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.913508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:13:24.994519 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994483    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:13:24.994819 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994806    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="storage-initializer"
Sep 18 20:13:24.994878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994821    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="storage-initializer"
Sep 18 20:13:24.994878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994838    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main"
Sep 18 20:13:24.994878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994846    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main"
Sep 18 20:13:24.994990 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.994897    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="745d0b5f-e7b7-4d2b-8422-feae0fbf610a" containerName="main"
Sep 18 20:13:24.997836 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:24.997814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.001441 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.001413    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lncpr\""
Sep 18 20:13:25.001565 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.001485    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:13:25.001630 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.001569    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:13:25.001925 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.001882    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-8rzx5\""
Sep 18 20:13:25.002110 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.002096    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:13:25.013546 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.013517    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:13:25.105793 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.105761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.105951 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.105842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lwt\" (UniqueName: \"kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.207307 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.207223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lwt\" (UniqueName: \"kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.207498 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.207322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.210065 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.210038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.217956 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.217930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lwt\" (UniqueName: \"kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.309138 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.309096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:25.597322 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:25.597178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:13:25.599918 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:13:25.599892    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb389b26_a842_4ed1_a04e_651263d4b320.slice/crio-51fe1d586983e1d90e5ff018f880773fe11ae6560c7c863c9ab78979ff7fae33 WatchSource:0}: Error finding container 51fe1d586983e1d90e5ff018f880773fe11ae6560c7c863c9ab78979ff7fae33: Status 404 returned error can't find the container with id 51fe1d586983e1d90e5ff018f880773fe11ae6560c7c863c9ab78979ff7fae33
Sep 18 20:13:26.130126 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:26.130085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" event={"ID":"bb389b26-a842-4ed1-a04e-651263d4b320","Type":"ContainerStarted","Data":"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"}
Sep 18 20:13:26.130517 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:26.130148    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" event={"ID":"bb389b26-a842-4ed1-a04e-651263d4b320","Type":"ContainerStarted","Data":"51fe1d586983e1d90e5ff018f880773fe11ae6560c7c863c9ab78979ff7fae33"}
Sep 18 20:13:26.130517 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:26.130232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:13:55.228947 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:55.228915    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-55c565b586-zghnx_f5fa2b40-bd32-40a0-9006-b53544cee4b1/storage-initializer/0.log"
Sep 18 20:13:55.241163 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:55.241136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-55c565b586-zghnx_f5fa2b40-bd32-40a0-9006-b53544cee4b1/main/0.log"
Sep 18 20:13:56.640945 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:56.640890    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podStartSLOduration=32.640870868 podStartE2EDuration="32.640870868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:26.148898899 +0000 UTC m=+3538.386025886" watchObservedRunningTime="2026-09-18 20:13:56.640870868 +0000 UTC m=+3568.877997855"
Sep 18 20:13:56.642828 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:56.642798    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:13:57.136950 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:13:57.136919    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:14:11.665791 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:11.665687    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" containerID="cri-o://03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57" gracePeriod=45
Sep 18 20:14:11.906535 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:11.906499    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 18 20:14:12.324715 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.324691    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:14:12.379816 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.379750    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerID="03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57" exitCode=0
Sep 18 20:14:12.379816 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.379787    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"
Sep 18 20:14:12.379969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.379820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerDied","Data":"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"}
Sep 18 20:14:12.379969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.379878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx" event={"ID":"f5fa2b40-bd32-40a0-9006-b53544cee4b1","Type":"ContainerDied","Data":"99e1f6d08ebfe9fe1733ca306c7d9b8d729fb312f706bf4337fd42ce582781a9"}
Sep 18 20:14:12.379969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.379907    2582 scope.go:117] "RemoveContainer" containerID="03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"
Sep 18 20:14:12.388457 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.388440    2582 scope.go:117] "RemoveContainer" containerID="426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"
Sep 18 20:14:12.394609 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394584    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-582sj\" (UniqueName: \"kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.394714 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.394714 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.394860 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394762    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.394860 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.394860 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.395025 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.395025 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.395025 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.394971    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert\") pod \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\" (UID: \"f5fa2b40-bd32-40a0-9006-b53544cee4b1\") "
Sep 18 20:14:12.395493 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.395441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:12.395591 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.395531    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:12.395871 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.395848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home" (OuterVolumeSpecName: "home") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:12.397079 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397013    2582 scope.go:117] "RemoveContainer" containerID="03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"
Sep 18 20:14:12.397778 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj" (OuterVolumeSpecName: "kube-api-access-582sj") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "kube-api-access-582sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:12.397778 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:14:12.397576    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57\": container with ID starting with 03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57 not found: ID does not exist" containerID="03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"
Sep 18 20:14:12.397778 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397609    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57"} err="failed to get container status \"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57\": rpc error: code = NotFound desc = could not find container \"03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57\": container with ID starting with 03b21317b73cd634e6f24be762c41ac3dc66e5b1943c1204a8c56be5771f5e57 not found: ID does not exist"
Sep 18 20:14:12.397778 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397633    2582 scope.go:117] "RemoveContainer" containerID="426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"
Sep 18 20:14:12.397778 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm" (OuterVolumeSpecName: "dshm") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:12.398098 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:14:12.397931    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd\": container with ID starting with 426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd not found: ID does not exist" containerID="426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"
Sep 18 20:14:12.398098 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397962    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd"} err="failed to get container status \"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd\": rpc error: code = NotFound desc = could not find container \"426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd\": container with ID starting with 426b62306342b71a948f82598d728db449698d9f5787e2cdbb78914b9c8501dd not found: ID does not exist"
Sep 18 20:14:12.398098 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.397938    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:12.400440 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.400421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 20:14:12.415392 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.415370    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:12.425005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.424984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5fa2b40-bd32-40a0-9006-b53544cee4b1" (UID: "f5fa2b40-bd32-40a0-9006-b53544cee4b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:12.495913 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495891    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.495913 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495913    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495922    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa2b40-bd32-40a0-9006-b53544cee4b1-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495930    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-582sj\" (UniqueName: \"kubernetes.io/projected/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kube-api-access-582sj\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495938    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495949    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495973    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") on node \"ip-10-0-143-75.ec2.internal\" "
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495982    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.496027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.495990    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5fa2b40-bd32-40a0-9006-b53544cee4b1-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:12.505123 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.505105    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492") on node "ip-10-0-143-75.ec2.internal"
Sep 18 20:14:12.597027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.597003    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-ca1b85ec-8035-45b5-8d18-0534abe901c6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a09d88d4c4627492\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"csi-559c9459636d88d22043d689b5f71bf416c602502cae5a7763f62b42a7515d0f\""
Sep 18 20:14:12.706836 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.706804    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:14:12.709954 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:12.709931    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-55c565b586-zghnx"]
Sep 18 20:14:13.894826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:13.894792    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" path="/var/lib/kubelet/pods/f5fa2b40-bd32-40a0-9006-b53544cee4b1/volumes"
Sep 18 20:14:20.264882 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.264844    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"]
Sep 18 20:14:20.265280 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.265155    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="storage-initializer"
Sep 18 20:14:20.265280 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.265169    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="storage-initializer"
Sep 18 20:14:20.265280 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.265187    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main"
Sep 18 20:14:20.265280 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.265192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main"
Sep 18 20:14:20.265280 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.265250    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fa2b40-bd32-40a0-9006-b53544cee4b1" containerName="main"
Sep 18 20:14:20.269865 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.269847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:20.276105 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.275952    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 18 20:14:20.278083 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.276583    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ns7xg\""
Sep 18 20:14:20.278083 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.277440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 20:14:20.278083 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.277929    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 20:14:20.291090 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.291063    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"]
Sep 18 20:14:20.455214 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.455175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cng25\" (UniqueName: \"kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:20.556310 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.556221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cng25\" (UniqueName: \"kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:20.577080 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:20.577052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cng25\" (UniqueName: \"kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:21.731949 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:21.731915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"]
Sep 18 20:14:21.738461 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:21.738439    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qgw2b"]
Sep 18 20:14:21.894712 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:21.894682    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa03d1ee-8233-420e-a15a-88c9dc4ef029" path="/var/lib/kubelet/pods/aa03d1ee-8233-420e-a15a-88c9dc4ef029/volumes"
Sep 18 20:14:27.908368 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:27.908337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:27.915183 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:27.915157    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.009014 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.008982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.009123 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.009095    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.012593 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.012576    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") DevicePath \"csi-5120c23666e92136be0e171615bbe30924d4589dc945d6b2076df206a228aef8\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.220445 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.220419    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f46428fd24186c3a205fc2c603f67c4c04b9d1392b3f5983d572cda5f8815967/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.234228 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.234199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"e2e-pvc-lora-download-n46hf\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.380150 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.380112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:28.653609 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:28.653583    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"]
Sep 18 20:14:28.655686 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:14:28.655656    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod212bd7bc_62ea_4d2e_a6e8_69eb3ef1c90e.slice/crio-5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077 WatchSource:0}: Error finding container 5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077: Status 404 returned error can't find the container with id 5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077
Sep 18 20:14:29.465527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:29.465480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf" event={"ID":"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e","Type":"ContainerStarted","Data":"0e8562a3889f7e9e5857eb9a3dd48535a8d9d60c8d419412e2ab19ec10b9b3ae"}
Sep 18 20:14:29.465527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:29.465543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf" event={"ID":"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e","Type":"ContainerStarted","Data":"5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077"}
Sep 18 20:14:29.499151 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:29.499061    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf" podStartSLOduration=12.499037933 podStartE2EDuration="12.499037933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:29.496089137 +0000 UTC m=+3601.733216124" watchObservedRunningTime="2026-09-18 20:14:29.499037933 +0000 UTC m=+3601.736164913"
Sep 18 20:14:30.117825 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:30.117797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:14:30.124950 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:30.124927    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:14:30.125845 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:30.125826    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:14:30.132796 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:30.132777    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:14:31.475927 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:31.475899    2582 generic.go:356] "Generic (PLEG): container finished" podID="212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" containerID="0e8562a3889f7e9e5857eb9a3dd48535a8d9d60c8d419412e2ab19ec10b9b3ae" exitCode=0
Sep 18 20:14:31.476355 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:31.475973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf" event={"ID":"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e","Type":"ContainerDied","Data":"0e8562a3889f7e9e5857eb9a3dd48535a8d9d60c8d419412e2ab19ec10b9b3ae"}
Sep 18 20:14:32.610940 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.610915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:32.748768 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.748676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") "
Sep 18 20:14:32.748917 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.748783    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cng25\" (UniqueName: \"kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25\") pod \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\" (UID: \"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e\") "
Sep 18 20:14:32.751216 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.751185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25" (OuterVolumeSpecName: "kube-api-access-cng25") pod "212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" (UID: "212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e"). InnerVolumeSpecName "kube-api-access-cng25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:32.751454 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.751431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8" (OuterVolumeSpecName: "model-storage") pod "212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" (UID: "212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e"). InnerVolumeSpecName "pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 20:14:32.850029 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.850000    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") on node \"ip-10-0-143-75.ec2.internal\" "
Sep 18 20:14:32.850029 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:32.850028    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cng25\" (UniqueName: \"kubernetes.io/projected/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e-kube-api-access-cng25\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:14:33.485683 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:33.485649    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"
Sep 18 20:14:33.485683 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:33.485667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf" event={"ID":"212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e","Type":"ContainerDied","Data":"5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077"}
Sep 18 20:14:33.485978 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:33.485705    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f90c7c1d6a7b03d7edb89db60e608986d6817f367bb0b492086fef525b82077"
Sep 18 20:14:33.822195 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:33.822174    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8") on node "ip-10-0-143-75.ec2.internal"
Sep 18 20:14:33.857408 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:14:33.857379    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"csi-5120c23666e92136be0e171615bbe30924d4589dc945d6b2076df206a228aef8\""
Sep 18 20:15:05.293183 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.293146    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:15:05.293686 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.293456    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" containerName="storage-initializer"
Sep 18 20:15:05.293686 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.293467    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" containerName="storage-initializer"
Sep 18 20:15:05.293686 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.293518    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" containerName="storage-initializer"
Sep 18 20:15:05.295920 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.295903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.298472 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.298451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ns7xg\""
Sep 18 20:15:05.298645 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.298623    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 20:15:05.298752 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.298718    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 18 20:15:05.298888 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.298871    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 20:15:05.313930 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.313884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:15:05.369153 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.369119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.369341 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.369161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.369341 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.369188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.369341 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.369276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.369341 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.369315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470443 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktjh\" (UniqueName: \"kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470633 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.470972 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.471019 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.470996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.471060 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.471049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.473049 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.473018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.473447 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.473425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.571789 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.571682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktjh\" (UniqueName: \"kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:05.582698 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:05.582671    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktjh\" (UniqueName: \"kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.395971 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.395933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.401837 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.401805    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.496384 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.496354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.496500 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.496463    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.499202 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.499184    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") DevicePath \"csi-5120c23666e92136be0e171615bbe30924d4589dc945d6b2076df206a228aef8\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.585645 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.585613    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f46428fd24186c3a205fc2c603f67c4c04b9d1392b3f5983d572cda5f8815967/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.606753 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.606715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.612017 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.611997    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:08.897038 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:08.897014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:15:08.898935 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:15:08.898906    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccd29d9_362c_410d_9001_bc17313b52f3.slice/crio-c8ab74b0f194fbe3578435dc4f826c21fe5cb7e64dda71e54fdb8f9e7b561544 WatchSource:0}: Error finding container c8ab74b0f194fbe3578435dc4f826c21fe5cb7e64dda71e54fdb8f9e7b561544: Status 404 returned error can't find the container with id c8ab74b0f194fbe3578435dc4f826c21fe5cb7e64dda71e54fdb8f9e7b561544
Sep 18 20:15:09.660672 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:09.660620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" event={"ID":"4ccd29d9-362c-410d-9001-bc17313b52f3","Type":"ContainerStarted","Data":"bdce22caeb72e302c933d0b6ded2426b6478e5ac3d2bc035403ba4fa361426e6"}
Sep 18 20:15:09.660672 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:09.660678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" event={"ID":"4ccd29d9-362c-410d-9001-bc17313b52f3","Type":"ContainerStarted","Data":"c8ab74b0f194fbe3578435dc4f826c21fe5cb7e64dda71e54fdb8f9e7b561544"}
Sep 18 20:15:09.683919 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:09.683859    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podStartSLOduration=4.683838861 podStartE2EDuration="4.683838861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:09.681580027 +0000 UTC m=+3641.918707014" watchObservedRunningTime="2026-09-18 20:15:09.683838861 +0000 UTC m=+3641.920965851"
Sep 18 20:15:10.612751 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:10.612693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:18.612170 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:18.612128    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:15:18.613719 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:18.613687    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:15:28.612686 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:28.612623    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:15:38.612894 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:38.612848    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:15:48.613156 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:48.613052    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:15:53.461012 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:53.460978    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:15:57.136530 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:57.136488    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:15:58.613267 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:15:58.613217    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:16:07.137051 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:07.137012    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:08.495869 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:08.495828    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" containerID="cri-o://53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015" gracePeriod=45
Sep 18 20:16:08.612809 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:08.612769    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:16:17.137221 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:17.137171    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:17.137611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:17.137342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:16:18.613436 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:18.613390    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 18 20:16:27.137054 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:27.137008    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:28.622086 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:28.622053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:16:28.629591 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:28.629566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:16:37.136966 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:37.136920    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:16:38.737852 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:38.737825    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:16:38.926580 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:38.926476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs\") pod \"bb389b26-a842-4ed1-a04e-651263d4b320\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") "
Sep 18 20:16:38.926580 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:38.926565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9lwt\" (UniqueName: \"kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt\") pod \"bb389b26-a842-4ed1-a04e-651263d4b320\" (UID: \"bb389b26-a842-4ed1-a04e-651263d4b320\") "
Sep 18 20:16:38.928892 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:38.928862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb389b26-a842-4ed1-a04e-651263d4b320" (UID: "bb389b26-a842-4ed1-a04e-651263d4b320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:38.928892 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:38.928864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt" (OuterVolumeSpecName: "kube-api-access-v9lwt") pod "bb389b26-a842-4ed1-a04e-651263d4b320" (UID: "bb389b26-a842-4ed1-a04e-651263d4b320"). InnerVolumeSpecName "kube-api-access-v9lwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:39.027464 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.027413    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lwt\" (UniqueName: \"kubernetes.io/projected/bb389b26-a842-4ed1-a04e-651263d4b320-kube-api-access-v9lwt\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:16:39.027464 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.027454    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb389b26-a842-4ed1-a04e-651263d4b320-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:16:39.128797 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.128758    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb389b26-a842-4ed1-a04e-651263d4b320" containerID="53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015" exitCode=0
Sep 18 20:16:39.128997 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.128857    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"
Sep 18 20:16:39.128997 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.128898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" event={"ID":"bb389b26-a842-4ed1-a04e-651263d4b320","Type":"ContainerDied","Data":"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"}
Sep 18 20:16:39.128997 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.128954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr" event={"ID":"bb389b26-a842-4ed1-a04e-651263d4b320","Type":"ContainerDied","Data":"51fe1d586983e1d90e5ff018f880773fe11ae6560c7c863c9ab78979ff7fae33"}
Sep 18 20:16:39.128997 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.128979    2582 scope.go:117] "RemoveContainer" containerID="53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"
Sep 18 20:16:39.138391 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.138370    2582 scope.go:117] "RemoveContainer" containerID="53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"
Sep 18 20:16:39.138630 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:16:39.138610    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015\": container with ID starting with 53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015 not found: ID does not exist" containerID="53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"
Sep 18 20:16:39.138678 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.138643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015"} err="failed to get container status \"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015\": rpc error: code = NotFound desc = could not find container \"53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015\": container with ID starting with 53d43cfd1a4fa526366dc7fa663572f6a3b78b9852b62fa4b8a628c693a60015 not found: ID does not exist"
Sep 18 20:16:39.152975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.152948    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:16:39.155974 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.155951    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pbnlr"]
Sep 18 20:16:39.895187 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:39.895142    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" path="/var/lib/kubelet/pods/bb389b26-a842-4ed1-a04e-651263d4b320/volumes"
Sep 18 20:16:54.622627 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:16:54.622593    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z_4ccd29d9-362c-410d-9001-bc17313b52f3/main/0.log"
Sep 18 20:17:09.779236 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:09.779152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:17:24.801740 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:24.801676    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main" containerID="cri-o://bdce22caeb72e302c933d0b6ded2426b6478e5ac3d2bc035403ba4fa361426e6" gracePeriod=45
Sep 18 20:17:25.366689 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.366655    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerID="bdce22caeb72e302c933d0b6ded2426b6478e5ac3d2bc035403ba4fa361426e6" exitCode=0
Sep 18 20:17:25.366868 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.366753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" event={"ID":"4ccd29d9-362c-410d-9001-bc17313b52f3","Type":"ContainerDied","Data":"bdce22caeb72e302c933d0b6ded2426b6478e5ac3d2bc035403ba4fa361426e6"}
Sep 18 20:17:25.451993 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.451963    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:17:25.607954 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.607868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608119 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.607988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608119 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608119 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608044    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fktjh\" (UniqueName: \"kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608119 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608338 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608338 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608160    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm\") pod \"4ccd29d9-362c-410d-9001-bc17313b52f3\" (UID: \"4ccd29d9-362c-410d-9001-bc17313b52f3\") "
Sep 18 20:17:25.608338 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608263    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:25.608489 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608401    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.608681 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.608653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home" (OuterVolumeSpecName: "home") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:25.610743 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.610701    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh" (OuterVolumeSpecName: "kube-api-access-fktjh") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "kube-api-access-fktjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:25.610988 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.610961    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:25.611086 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.611046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm" (OuterVolumeSpecName: "dshm") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:25.616196 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.616176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8" (OuterVolumeSpecName: "kserve-pvc-source") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 20:17:25.619612 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.619589    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ccd29d9-362c-410d-9001-bc17313b52f3" (UID: "4ccd29d9-362c-410d-9001-bc17313b52f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:25.708975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.708948    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") on node \"ip-10-0-143-75.ec2.internal\" "
Sep 18 20:17:25.708975 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.708973    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktjh\" (UniqueName: \"kubernetes.io/projected/4ccd29d9-362c-410d-9001-bc17313b52f3-kube-api-access-fktjh\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.709137 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.708983    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.709137 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.708993    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccd29d9-362c-410d-9001-bc17313b52f3-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.709137 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.709001    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.709137 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.709009    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccd29d9-362c-410d-9001-bc17313b52f3-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:17:25.738675 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.738659    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8") on node "ip-10-0-143-75.ec2.internal"
Sep 18 20:17:25.809576 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:25.809553    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-bdc5ec2d-60c8-4877-a83f-72c0e763eef9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ff220d96843366a8\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"csi-5120c23666e92136be0e171615bbe30924d4589dc945d6b2076df206a228aef8\""
Sep 18 20:17:26.373339 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:26.373306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"
Sep 18 20:17:26.373533 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:26.373347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z" event={"ID":"4ccd29d9-362c-410d-9001-bc17313b52f3","Type":"ContainerDied","Data":"c8ab74b0f194fbe3578435dc4f826c21fe5cb7e64dda71e54fdb8f9e7b561544"}
Sep 18 20:17:26.373533 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:26.373415    2582 scope.go:117] "RemoveContainer" containerID="bdce22caeb72e302c933d0b6ded2426b6478e5ac3d2bc035403ba4fa361426e6"
Sep 18 20:17:26.396324 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:26.396299    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:17:26.402254 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:26.402233    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-6cd5z"]
Sep 18 20:17:27.895114 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:27.895079    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" path="/var/lib/kubelet/pods/4ccd29d9-362c-410d-9001-bc17313b52f3/volumes"
Sep 18 20:17:35.774373 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:35.774336    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"]
Sep 18 20:17:35.778190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:35.778167    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-n46hf"]
Sep 18 20:17:35.895044 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:35.895010    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e" path="/var/lib/kubelet/pods/212bd7bc-62ea-4d2e-a6e8-69eb3ef1c90e/volumes"
Sep 18 20:17:37.214706 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.214678    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.214977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main"
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.214990    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main"
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.215001    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main"
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.215007    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main"
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.215065    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ccd29d9-362c-410d-9001-bc17313b52f3" containerName="main"
Sep 18 20:17:37.215154 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.215073    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb389b26-a842-4ed1-a04e-651263d4b320" containerName="main"
Sep 18 20:17:37.217830 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.217815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.220235 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:17:37.220413 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220397    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 18 20:17:37.220501 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220473    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 20:17:37.220501 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-6z5xm\""
Sep 18 20:17:37.220611 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220546    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 20:17:37.220774 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.220756    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 20:17:37.266399 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.266373    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:17:37.398919 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.398887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.398919 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.398922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399118 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.398956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljfj\" (UniqueName: \"kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399118 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.399025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399118 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.399076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399118 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.399110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399255 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.399140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.399255 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.399158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.483481 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.483402    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:17:37.486399 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.486378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.488941 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.488919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-vrv9w\""
Sep 18 20:17:37.496722 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.496703    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:17:37.500309 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500408 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500408 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500556 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500634 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500634 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500769 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500769 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljfj\" (UniqueName: \"kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500769 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500687    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500769 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.500938 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.500824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.501346 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.501281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.501346 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.501311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.503033 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.503014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.503289 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.503274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.519151 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.519122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljfj\" (UniqueName: \"kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj\") pod \"tls-verification-test-kserve-86d7fc5986-wvd9h\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.527490 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.527471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:37.601378 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.601344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.601501 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.601436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r2z\" (UniqueName: \"kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.702566 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.702525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.702766 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.702623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r2z\" (UniqueName: \"kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.705191 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.705169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.710788 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.710771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r2z\" (UniqueName: \"kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.796889 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.796808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:37.816576 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:37.816548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:17:37.820214 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:17:37.820187    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2561cb4b_68bd_4399_92ca_92127085ef93.slice/crio-f6c55818631e43386e966241e7ca35cbee6ea0847b8b700dfc05394ee84d18c1 WatchSource:0}: Error finding container f6c55818631e43386e966241e7ca35cbee6ea0847b8b700dfc05394ee84d18c1: Status 404 returned error can't find the container with id f6c55818631e43386e966241e7ca35cbee6ea0847b8b700dfc05394ee84d18c1
Sep 18 20:17:38.085418 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.085385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:17:38.087399 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:17:38.087369    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8005bd_bc12_4122_ba38_962687e1d482.slice/crio-58e3c89288c39c02855881e02ca3f29d07b21a98853f2c2a05d921b1e4266d16 WatchSource:0}: Error finding container 58e3c89288c39c02855881e02ca3f29d07b21a98853f2c2a05d921b1e4266d16: Status 404 returned error can't find the container with id 58e3c89288c39c02855881e02ca3f29d07b21a98853f2c2a05d921b1e4266d16
Sep 18 20:17:38.431763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.431651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" event={"ID":"3f8005bd-bc12-4122-ba38-962687e1d482","Type":"ContainerStarted","Data":"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"}
Sep 18 20:17:38.431763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.431743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" event={"ID":"3f8005bd-bc12-4122-ba38-962687e1d482","Type":"ContainerStarted","Data":"58e3c89288c39c02855881e02ca3f29d07b21a98853f2c2a05d921b1e4266d16"}
Sep 18 20:17:38.432319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.431778    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:17:38.433323 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.433289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerStarted","Data":"3f6464902a1c61b508a057797c410796de4333bfa98c6257ab6c5b3d6e6da9e3"}
Sep 18 20:17:38.433453 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.433332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerStarted","Data":"f6c55818631e43386e966241e7ca35cbee6ea0847b8b700dfc05394ee84d18c1"}
Sep 18 20:17:38.512349 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:38.512297    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podStartSLOduration=1.512279903 podStartE2EDuration="1.512279903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:38.465777153 +0000 UTC m=+3790.702904141" watchObservedRunningTime="2026-09-18 20:17:38.512279903 +0000 UTC m=+3790.749406888"
Sep 18 20:17:40.446066 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:40.445976    2582 generic.go:356] "Generic (PLEG): container finished" podID="2561cb4b-68bd-4399-92ca-92127085ef93" containerID="3f6464902a1c61b508a057797c410796de4333bfa98c6257ab6c5b3d6e6da9e3" exitCode=0
Sep 18 20:17:40.446066 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:40.446048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerDied","Data":"3f6464902a1c61b508a057797c410796de4333bfa98c6257ab6c5b3d6e6da9e3"}
Sep 18 20:17:41.453916 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:41.453880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerStarted","Data":"7c03ad58cd79e3ea2492deacf0c3c2dd3cf0880f3749b69156f44d95bf56bba0"}
Sep 18 20:17:41.483139 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:41.483087    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podStartSLOduration=2.278450722 podStartE2EDuration="4.483070052s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.20461933s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:41.48283468 +0000 UTC m=+3793.719961667" watchObservedRunningTime="2026-09-18 20:17:41.483070052 +0000 UTC m=+3793.720197038"
Sep 18 20:17:41.527711 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:41.527678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:47.528256 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:47.528217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:17:47.530032 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:47.529997    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:17:57.528951 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:17:57.528908    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:07.528711 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:07.528665    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:09.441081 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:09.441053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:18:17.528103 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:17.528064    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:27.528135 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:27.528088    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:30.379308 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:30.379276    2582 scope.go:117] "RemoveContainer" containerID="78a8751541bf9f9e9ea22e39aa9392f709a5ea69276db8d80c7f3319cb53226f"
Sep 18 20:18:37.528099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:37.528051    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:47.529027 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:47.528923    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:18:57.528887 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:18:57.528845    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:19:07.528819 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:07.528772    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:19:17.528870 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:17.528825    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 18 20:19:27.538120 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:27.538084    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:19:27.545763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:27.545721    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:19:30.157237 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:30.157202    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:19:30.164324 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:30.164300    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:19:30.166490 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:30.166470    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:19:30.173522 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:30.173503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:19:59.655044 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:59.655006    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:19:59.660361 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:19:59.660329    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:20:09.441914 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:09.441834    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:20:14.679199 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:14.679157    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main" containerID="cri-o://7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553" gracePeriod=45
Sep 18 20:20:14.679597 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:14.679209    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main" containerID="cri-o://7c03ad58cd79e3ea2492deacf0c3c2dd3cf0880f3749b69156f44d95bf56bba0" gracePeriod=45
Sep 18 20:20:15.232440 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.232413    2582 generic.go:356] "Generic (PLEG): container finished" podID="2561cb4b-68bd-4399-92ca-92127085ef93" containerID="7c03ad58cd79e3ea2492deacf0c3c2dd3cf0880f3749b69156f44d95bf56bba0" exitCode=0
Sep 18 20:20:15.232572 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.232485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerDied","Data":"7c03ad58cd79e3ea2492deacf0c3c2dd3cf0880f3749b69156f44d95bf56bba0"}
Sep 18 20:20:15.348832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.348809    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:20:15.474056 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sljfj\" (UniqueName: \"kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474188    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474524 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474241    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm\") pod \"2561cb4b-68bd-4399-92ca-92127085ef93\" (UID: \"2561cb4b-68bd-4399-92ca-92127085ef93\") "
Sep 18 20:20:15.474524 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache" (OuterVolumeSpecName: "model-cache") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:15.474971 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:20:15.474971 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.474912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home" (OuterVolumeSpecName: "home") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:15.476529 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.476506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:20:15.476646 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.476626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm" (OuterVolumeSpecName: "dshm") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:15.476863 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.476843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj" (OuterVolumeSpecName: "kube-api-access-sljfj") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "kube-api-access-sljfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:20:15.487275 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.487250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:15.504878 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.504852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2561cb4b-68bd-4399-92ca-92127085ef93" (UID: "2561cb4b-68bd-4399-92ca-92127085ef93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:15.575121 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575095    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575121 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575121    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575130    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljfj\" (UniqueName: \"kubernetes.io/projected/2561cb4b-68bd-4399-92ca-92127085ef93-kube-api-access-sljfj\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575139    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575147    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575155    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2561cb4b-68bd-4399-92ca-92127085ef93-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575162    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2561cb4b-68bd-4399-92ca-92127085ef93-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:15.575261 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:15.575170    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2561cb4b-68bd-4399-92ca-92127085ef93-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:16.239205 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.239180    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"
Sep 18 20:20:16.239621 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.239174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h" event={"ID":"2561cb4b-68bd-4399-92ca-92127085ef93","Type":"ContainerDied","Data":"f6c55818631e43386e966241e7ca35cbee6ea0847b8b700dfc05394ee84d18c1"}
Sep 18 20:20:16.239779 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.239657    2582 scope.go:117] "RemoveContainer" containerID="7c03ad58cd79e3ea2492deacf0c3c2dd3cf0880f3749b69156f44d95bf56bba0"
Sep 18 20:20:16.248620 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.248602    2582 scope.go:117] "RemoveContainer" containerID="3f6464902a1c61b508a057797c410796de4333bfa98c6257ab6c5b3d6e6da9e3"
Sep 18 20:20:16.261228 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.261208    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:20:16.267788 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:16.267765    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-86d7fc5986-wvd9h"]
Sep 18 20:20:17.895334 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:17.895298    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" path="/var/lib/kubelet/pods/2561cb4b-68bd-4399-92ca-92127085ef93/volumes"
Sep 18 20:20:19.441368 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:19.441323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:20:29.441279 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:29.441242    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:20:29.441681 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:29.441336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:20:30.406645 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:30.406607    2582 scope.go:117] "RemoveContainer" containerID="0e8562a3889f7e9e5857eb9a3dd48535a8d9d60c8d419412e2ab19ec10b9b3ae"
Sep 18 20:20:39.440968 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:39.440931    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:20:44.926520 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:44.926490    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:20:44.995005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:44.994978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs\") pod \"3f8005bd-bc12-4122-ba38-962687e1d482\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") "
Sep 18 20:20:44.995141 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:44.995072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55r2z\" (UniqueName: \"kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z\") pod \"3f8005bd-bc12-4122-ba38-962687e1d482\" (UID: \"3f8005bd-bc12-4122-ba38-962687e1d482\") "
Sep 18 20:20:44.997135 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:44.997102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f8005bd-bc12-4122-ba38-962687e1d482" (UID: "3f8005bd-bc12-4122-ba38-962687e1d482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:20:44.997249 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:44.997198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z" (OuterVolumeSpecName: "kube-api-access-55r2z") pod "3f8005bd-bc12-4122-ba38-962687e1d482" (UID: "3f8005bd-bc12-4122-ba38-962687e1d482"). InnerVolumeSpecName "kube-api-access-55r2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:20:45.096430 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.096396    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8005bd-bc12-4122-ba38-962687e1d482-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.096430 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.096430    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55r2z\" (UniqueName: \"kubernetes.io/projected/3f8005bd-bc12-4122-ba38-962687e1d482-kube-api-access-55r2z\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:20:45.378845 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.378761    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f8005bd-bc12-4122-ba38-962687e1d482" containerID="7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553" exitCode=0
Sep 18 20:20:45.378845 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.378805    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"
Sep 18 20:20:45.379022 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.378837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" event={"ID":"3f8005bd-bc12-4122-ba38-962687e1d482","Type":"ContainerDied","Data":"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"}
Sep 18 20:20:45.379022 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.378886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk" event={"ID":"3f8005bd-bc12-4122-ba38-962687e1d482","Type":"ContainerDied","Data":"58e3c89288c39c02855881e02ca3f29d07b21a98853f2c2a05d921b1e4266d16"}
Sep 18 20:20:45.379022 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.378908    2582 scope.go:117] "RemoveContainer" containerID="7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"
Sep 18 20:20:45.387788 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.387771    2582 scope.go:117] "RemoveContainer" containerID="7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"
Sep 18 20:20:45.388050 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:20:45.388031    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553\": container with ID starting with 7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553 not found: ID does not exist" containerID="7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"
Sep 18 20:20:45.388099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.388059    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553"} err="failed to get container status \"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553\": rpc error: code = NotFound desc = could not find container \"7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553\": container with ID starting with 7fa86190ba44796e4c238e015978bdfab0f9096ec817a31eda1b164f149de553 not found: ID does not exist"
Sep 18 20:20:45.400315 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.400292    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:20:45.403473 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.403452    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-mz4zk"]
Sep 18 20:20:45.894392 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:20:45.894363    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" path="/var/lib/kubelet/pods/3f8005bd-bc12-4122-ba38-962687e1d482/volumes"
Sep 18 20:21:01.283576 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.283533    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.283950    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="storage-initializer"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.283970    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="storage-initializer"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284004    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284012    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284023    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284030    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main"
Sep 18 20:21:01.284101 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284102    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2561cb4b-68bd-4399-92ca-92127085ef93" containerName="main"
Sep 18 20:21:01.284463 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.284112    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8005bd-bc12-4122-ba38-962687e1d482" containerName="main"
Sep 18 20:21:01.288553 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.288530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.291719 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291696    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 20:21:01.291839 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291700    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:21:01.291839 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 20:21:01.291969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291712    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 20:21:01.291969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291715    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 20:21:01.291969 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.291779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-b9l6g\""
Sep 18 20:21:01.305666 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.305638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:21:01.411207 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411207 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411466 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411646 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrq9v\" (UniqueName: \"kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.411646 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.411498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512094 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512299 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512299 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq9v\" (UniqueName: \"kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512299 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512299 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512515 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512515 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512606 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512606 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512815 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512924 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.512924 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.512866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.513097 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.513080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.514803 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.514772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.515111 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.515093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.520858 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.520835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq9v\" (UniqueName: \"kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v\") pod \"prestop-hook-test-kserve-7db57ffd58-t7v7z\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.598047 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.597975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:01.713999 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.713966    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:21:01.717350 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.717324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.719770 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.719719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-84x2l\""
Sep 18 20:21:01.728206 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.728186    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:21:01.815039 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.815001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blx6t\" (UniqueName: \"kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.815190 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.815053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.897356 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.897284    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:21:01.900371 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:21:01.900339    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e2bdea_530c_46cb_830e_328ba913f219.slice/crio-eb08f1dddb3be19be07a230d1efa2f8ecb777564c05cbb6e289a3b85ceb03e02 WatchSource:0}: Error finding container eb08f1dddb3be19be07a230d1efa2f8ecb777564c05cbb6e289a3b85ceb03e02: Status 404 returned error can't find the container with id eb08f1dddb3be19be07a230d1efa2f8ecb777564c05cbb6e289a3b85ceb03e02
Sep 18 20:21:01.916414 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.916396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blx6t\" (UniqueName: \"kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.916504 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.916433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.919049 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.919026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:01.931065 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:01.930908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blx6t\" (UniqueName: \"kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:02.027520 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.027460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:02.333447 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.333421    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:21:02.335487 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:21:02.335447    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40513858_37c9_4ce6_a27e_8d7287666d4d.slice/crio-96caa63c9563b3c5dc8fb3009102a5ec5408baa47d9632e23e69e5a99c4d0ef6 WatchSource:0}: Error finding container 96caa63c9563b3c5dc8fb3009102a5ec5408baa47d9632e23e69e5a99c4d0ef6: Status 404 returned error can't find the container with id 96caa63c9563b3c5dc8fb3009102a5ec5408baa47d9632e23e69e5a99c4d0ef6
Sep 18 20:21:02.471440 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.471398    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerStarted","Data":"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"}
Sep 18 20:21:02.471680 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.471500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerStarted","Data":"eb08f1dddb3be19be07a230d1efa2f8ecb777564c05cbb6e289a3b85ceb03e02"}
Sep 18 20:21:02.473313 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.473280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" event={"ID":"40513858-37c9-4ce6-a27e-8d7287666d4d","Type":"ContainerStarted","Data":"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"}
Sep 18 20:21:02.473420 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.473340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" event={"ID":"40513858-37c9-4ce6-a27e-8d7287666d4d","Type":"ContainerStarted","Data":"96caa63c9563b3c5dc8fb3009102a5ec5408baa47d9632e23e69e5a99c4d0ef6"}
Sep 18 20:21:02.473420 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.473382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:02.520660 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:02.520589    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podStartSLOduration=1.520568296 podStartE2EDuration="1.520568296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:02.517113718 +0000 UTC m=+3994.754240706" watchObservedRunningTime="2026-09-18 20:21:02.520568296 +0000 UTC m=+3994.757695282"
Sep 18 20:21:04.485531 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:04.485499    2582 generic.go:356] "Generic (PLEG): container finished" podID="f1e2bdea-530c-46cb-830e-328ba913f219" containerID="caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121" exitCode=0
Sep 18 20:21:04.485971 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:04.485558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerDied","Data":"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"}
Sep 18 20:21:05.491881 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:05.491846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerStarted","Data":"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"}
Sep 18 20:21:05.517073 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:05.517021    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podStartSLOduration=2.33911874 podStartE2EDuration="4.517005649s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.177886909s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:05.51613015 +0000 UTC m=+3997.753257139" watchObservedRunningTime="2026-09-18 20:21:05.517005649 +0000 UTC m=+3997.754132663"
Sep 18 20:21:05.598507 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:05.598469    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:11.598618 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:11.598569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:21:11.599964 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:11.599935    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:21:21.599200 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:21.599145    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:21:31.599141 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:31.599083    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:21:33.481598 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:33.481566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:21:41.599409 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:41.599317    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:21:51.598542 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:21:51.598498    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:01.598797 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:01.598747    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:11.599149 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:11.599103    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:21.598918 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:21.598870    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:31.598655 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:31.598599    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:41.599163 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:41.599119    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 18 20:22:51.608459 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:51.608426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:22:51.616051 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:51.616028    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:22:55.921715 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:22:55.921679    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:23:10.944206 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:10.944117    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" containerID="cri-o://fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037" gracePeriod=45
Sep 18 20:23:11.615589 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.615564    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:23:11.752392 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752363    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752481    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrq9v\" (UniqueName: \"kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752568 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location\") pod \"f1e2bdea-530c-46cb-830e-328ba913f219\" (UID: \"f1e2bdea-530c-46cb-830e-328ba913f219\") "
Sep 18 20:23:11.752999 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.752936    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:11.753074 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.753040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:11.753172 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.753143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home" (OuterVolumeSpecName: "home") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:11.754894 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.754871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm" (OuterVolumeSpecName: "dshm") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:11.755128 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.755052    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v" (OuterVolumeSpecName: "kube-api-access-hrq9v") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "kube-api-access-hrq9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:11.755203 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.755150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:11.764836 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.764811    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:11.777771 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.777742    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1e2bdea-530c-46cb-830e-328ba913f219" (UID: "f1e2bdea-530c-46cb-830e-328ba913f219"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:11.853673 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853648    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e2bdea-530c-46cb-830e-328ba913f219-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853673 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853671    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-kserve-provision-location\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853680    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853688    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853715    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853723    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e2bdea-530c-46cb-830e-328ba913f219-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853753    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1e2bdea-530c-46cb-830e-328ba913f219-cabundle-cert\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:11.853826 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:11.853764    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrq9v\" (UniqueName: \"kubernetes.io/projected/f1e2bdea-530c-46cb-830e-328ba913f219-kube-api-access-hrq9v\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:23:12.113468 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.113396    2582 generic.go:356] "Generic (PLEG): container finished" podID="f1e2bdea-530c-46cb-830e-328ba913f219" containerID="fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037" exitCode=0
Sep 18 20:23:12.113468 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.113450    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"
Sep 18 20:23:12.113930 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.113481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerDied","Data":"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"}
Sep 18 20:23:12.113930 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.113537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" event={"ID":"f1e2bdea-530c-46cb-830e-328ba913f219","Type":"ContainerDied","Data":"eb08f1dddb3be19be07a230d1efa2f8ecb777564c05cbb6e289a3b85ceb03e02"}
Sep 18 20:23:12.113930 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.113561    2582 scope.go:117] "RemoveContainer" containerID="fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"
Sep 18 20:23:12.122560 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.122541    2582 scope.go:117] "RemoveContainer" containerID="caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"
Sep 18 20:23:12.130433 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.130412    2582 scope.go:117] "RemoveContainer" containerID="fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"
Sep 18 20:23:12.130773 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:23:12.130722    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037\": container with ID starting with fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037 not found: ID does not exist" containerID="fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"
Sep 18 20:23:12.130874 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.130775    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037"} err="failed to get container status \"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037\": rpc error: code = NotFound desc = could not find container \"fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037\": container with ID starting with fec4759359e24fea5b444143b6d4debfdd0ca1645611264e5168f6808e32b037 not found: ID does not exist"
Sep 18 20:23:12.130874 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.130796    2582 scope.go:117] "RemoveContainer" containerID="caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"
Sep 18 20:23:12.131056 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:23:12.131039    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121\": container with ID starting with caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121 not found: ID does not exist" containerID="caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"
Sep 18 20:23:12.131105 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.131060    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121"} err="failed to get container status \"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121\": rpc error: code = NotFound desc = could not find container \"caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121\": container with ID starting with caadfc0dc3ecd9546e2c066bfed6585341d42cc20d75b86d0402b9c7a6d6a121 not found: ID does not exist"
Sep 18 20:23:12.132790 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.132768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:23:12.137131 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.137113    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z"]
Sep 18 20:23:12.610376 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:12.610336    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7db57ffd58-t7v7z" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Sep 18 20:23:13.895216 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:13.895177    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" path="/var/lib/kubelet/pods/f1e2bdea-530c-46cb-830e-328ba913f219/volumes"
Sep 18 20:23:15.219918 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:15.219881    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:23:23.481212 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:23.481167    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:30.239296 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:30.239254    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main" containerID="cri-o://bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659" gracePeriod=45
Sep 18 20:23:33.481230 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:33.481192    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:43.480661 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:43.480623    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:23:43.481098 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:43.480755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:23:53.481142 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:23:53.481103    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:24:00.473388 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.473363    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:24:00.507985 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.507928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs\") pod \"40513858-37c9-4ce6-a27e-8d7287666d4d\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") "
Sep 18 20:24:00.507985 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.507969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blx6t\" (UniqueName: \"kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t\") pod \"40513858-37c9-4ce6-a27e-8d7287666d4d\" (UID: \"40513858-37c9-4ce6-a27e-8d7287666d4d\") "
Sep 18 20:24:00.510136 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.510107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40513858-37c9-4ce6-a27e-8d7287666d4d" (UID: "40513858-37c9-4ce6-a27e-8d7287666d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:00.510233 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.510180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t" (OuterVolumeSpecName: "kube-api-access-blx6t") pod "40513858-37c9-4ce6-a27e-8d7287666d4d" (UID: "40513858-37c9-4ce6-a27e-8d7287666d4d"). InnerVolumeSpecName "kube-api-access-blx6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:00.609221 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.609196    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40513858-37c9-4ce6-a27e-8d7287666d4d-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:24:00.609221 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:00.609218    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blx6t\" (UniqueName: \"kubernetes.io/projected/40513858-37c9-4ce6-a27e-8d7287666d4d-kube-api-access-blx6t\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:24:01.339014 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.338979    2582 generic.go:356] "Generic (PLEG): container finished" podID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerID="bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659" exitCode=0
Sep 18 20:24:01.339198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.339032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" event={"ID":"40513858-37c9-4ce6-a27e-8d7287666d4d","Type":"ContainerDied","Data":"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"}
Sep 18 20:24:01.339198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.339040    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"
Sep 18 20:24:01.339198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.339075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs" event={"ID":"40513858-37c9-4ce6-a27e-8d7287666d4d","Type":"ContainerDied","Data":"96caa63c9563b3c5dc8fb3009102a5ec5408baa47d9632e23e69e5a99c4d0ef6"}
Sep 18 20:24:01.339198 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.339095    2582 scope.go:117] "RemoveContainer" containerID="bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"
Sep 18 20:24:01.348817 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.348797    2582 scope.go:117] "RemoveContainer" containerID="bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"
Sep 18 20:24:01.349076 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:24:01.349056    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659\": container with ID starting with bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659 not found: ID does not exist" containerID="bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"
Sep 18 20:24:01.349124 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.349085    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659"} err="failed to get container status \"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659\": rpc error: code = NotFound desc = could not find container \"bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659\": container with ID starting with bf5b3a896a47b0abf1a13c9bfa159f743c3da98f813a736d0eebd943ce1d0659 not found: ID does not exist"
Sep 18 20:24:01.362747 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.362710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:24:01.369085 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.369063    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln2gs"]
Sep 18 20:24:01.894235 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:01.894203    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" path="/var/lib/kubelet/pods/40513858-37c9-4ce6-a27e-8d7287666d4d/volumes"
Sep 18 20:24:24.291010 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.290974    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291244    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291255    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291272    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="storage-initializer"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291277    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="storage-initializer"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291288    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291294    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291360    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e2bdea-530c-46cb-830e-328ba913f219" containerName="main"
Sep 18 20:24:24.291523 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.291369    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40513858-37c9-4ce6-a27e-8d7287666d4d" containerName="main"
Sep 18 20:24:24.294241 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.294214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.297336 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.297309    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 20:24:24.298139 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.298116    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 20:24:24.298249 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.298195    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 20:24:24.298249 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.298126    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-p2kjg\""
Sep 18 20:24:24.305325 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.305303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:24:24.376904 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.376873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.377042 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.376911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwfc\" (UniqueName: \"kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.377042 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.376930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.377042 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.376947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.377042 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.377022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.377211 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.377079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.477924 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.477895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.477928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.477953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.477987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.478013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478053 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.478033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwfc\" (UniqueName: \"kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.478366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478456 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.478377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.478456 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.478434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.480524 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.480497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.480793 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.480777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.485882 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.485866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwfc\" (UniqueName: \"kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc\") pod \"rolling-upgrade-test-kserve-74b79d7f77-wm6nd\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.519441 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.519394    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:24:24.526220 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.526198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.528569 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.528549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-qhrzk\""
Sep 18 20:24:24.536039 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.536019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:24:24.605347 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.605275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:24.680416 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.680387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2kx\" (UniqueName: \"kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.680549 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.680435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.781537 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.781501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.781684 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.781571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2kx\" (UniqueName: \"kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.784256 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.784231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.794021 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.793992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2kx\" (UniqueName: \"kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.837203 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.837176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:24.886601 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:24.886535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:24:24.889326 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:24:24.889292    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873e6ff5_c0b8_4962_9fb4_c31b3fafe629.slice/crio-f7e1d32ea8b2b04c8d12beee5a1fa6687600d1eb08921272d7b8c66849056b0d WatchSource:0}: Error finding container f7e1d32ea8b2b04c8d12beee5a1fa6687600d1eb08921272d7b8c66849056b0d: Status 404 returned error can't find the container with id f7e1d32ea8b2b04c8d12beee5a1fa6687600d1eb08921272d7b8c66849056b0d
Sep 18 20:24:25.120504 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.120446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:24:25.123133 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:24:25.123098    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d6b638_24ff_402a_9eff_a8f5f7e0d2c9.slice/crio-4b1cb48c3d820d1cd69206d2df4a96fede2e7b59be0c494b8ead4547a3f88c1e WatchSource:0}: Error finding container 4b1cb48c3d820d1cd69206d2df4a96fede2e7b59be0c494b8ead4547a3f88c1e: Status 404 returned error can't find the container with id 4b1cb48c3d820d1cd69206d2df4a96fede2e7b59be0c494b8ead4547a3f88c1e
Sep 18 20:24:25.456281 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.456196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" event={"ID":"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9","Type":"ContainerStarted","Data":"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"}
Sep 18 20:24:25.456281 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.456250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" event={"ID":"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9","Type":"ContainerStarted","Data":"4b1cb48c3d820d1cd69206d2df4a96fede2e7b59be0c494b8ead4547a3f88c1e"}
Sep 18 20:24:25.456817 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.456312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:24:25.457854 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.457829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" event={"ID":"873e6ff5-c0b8-4962-9fb4-c31b3fafe629","Type":"ContainerStarted","Data":"280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca"}
Sep 18 20:24:25.457939 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.457874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" event={"ID":"873e6ff5-c0b8-4962-9fb4-c31b3fafe629","Type":"ContainerStarted","Data":"f7e1d32ea8b2b04c8d12beee5a1fa6687600d1eb08921272d7b8c66849056b0d"}
Sep 18 20:24:25.476294 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.476249    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podStartSLOduration=1.476235797 podStartE2EDuration="1.476235797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:25.474070833 +0000 UTC m=+4197.711197820" watchObservedRunningTime="2026-09-18 20:24:25.476235797 +0000 UTC m=+4197.713362783"
Sep 18 20:24:25.493766 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.493708    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" podStartSLOduration=1.493695567 podStartE2EDuration="1.493695567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:25.491580775 +0000 UTC m=+4197.728707772" watchObservedRunningTime="2026-09-18 20:24:25.493695567 +0000 UTC m=+4197.730822552"
Sep 18 20:24:25.605600 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:25.605574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:30.192763 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:30.192713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:24:30.199390 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:30.199369    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:24:30.202718 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:30.202697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:24:30.208833 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:30.208817    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wlzg5_45c34a65-61c3-418d-8599-87cb5547cba6/ovn-acl-logging/0.log"
Sep 18 20:24:34.606048 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:34.606019    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:34.618822 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:34.618796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:35.515092 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:35.515061    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:24:56.465529 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:24:56.465438    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:25:07.051194 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.051157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:25:07.060085 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:25:07.060040    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:25:07.060085 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.060085 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.060085 ip-10-0-143-75 kubenswrapper[2582]:  > containerID="280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca" cmd=["/bin/sleep","15"]
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.060111    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" message="" err=<
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:  >
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:25:07.060127    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main" containerID="cri-o://280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca"
Sep 18 20:25:07.060375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.060200    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main" containerID="cri-o://280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca" gracePeriod=60
Sep 18 20:25:07.075328 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.075302    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:25:07.504226 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.504183    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 18 20:25:07.670547 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.670514    2582 generic.go:356] "Generic (PLEG): container finished" podID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerID="280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca" exitCode=0
Sep 18 20:25:07.670710 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.670556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" event={"ID":"873e6ff5-c0b8-4962-9fb4-c31b3fafe629","Type":"ContainerDied","Data":"280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca"}
Sep 18 20:25:07.799662 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.799634    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:25:07.898840 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.898794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:07.899024 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.898890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:07.899024 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.898953    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvwfc\" (UniqueName: \"kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:07.899238 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:07.899302 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:07.899404 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.899509 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home" (OuterVolumeSpecName: "home") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.899604 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899587    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tmp-dir\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.899666 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899610    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-home\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:07.899666 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.899613    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache" (OuterVolumeSpecName: "model-cache") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:07.901509 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.901482    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:07.901642 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:07.901622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc" (OuterVolumeSpecName: "kube-api-access-hvwfc") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "kube-api-access-hvwfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:08.000345 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.000315    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm\") pod \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\" (UID: \"873e6ff5-c0b8-4962-9fb4-c31b3fafe629\") "
Sep 18 20:25:08.000484 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.000428    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.000484 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.000439    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-model-cache\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.000484 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.000447    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvwfc\" (UniqueName: \"kubernetes.io/projected/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-kube-api-access-hvwfc\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.002389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.002365    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm" (OuterVolumeSpecName: "dshm") pod "873e6ff5-c0b8-4962-9fb4-c31b3fafe629" (UID: "873e6ff5-c0b8-4962-9fb4-c31b3fafe629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:08.101353 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.101289    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873e6ff5-c0b8-4962-9fb4-c31b3fafe629-dshm\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.677296 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.677261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd" event={"ID":"873e6ff5-c0b8-4962-9fb4-c31b3fafe629","Type":"ContainerDied","Data":"f7e1d32ea8b2b04c8d12beee5a1fa6687600d1eb08921272d7b8c66849056b0d"}
Sep 18 20:25:08.677491 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.677306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"
Sep 18 20:25:08.677491 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.677329    2582 scope.go:117] "RemoveContainer" containerID="280eb82d5440797f5d4bc8d6fcbe3e9d005d90f325d32e060270c84585c415ca"
Sep 18 20:25:08.708139 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.708113    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:25:08.720063 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:08.720041    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-wm6nd"]
Sep 18 20:25:09.895742 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:09.895679    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" path="/var/lib/kubelet/pods/873e6ff5-c0b8-4962-9fb4-c31b3fafe629/volumes"
Sep 18 20:25:16.465492 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:16.465449    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:22.094029 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:22.093985    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main" containerID="cri-o://7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830" gracePeriod=45
Sep 18 20:25:26.465065 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:26.465027    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:36.465232 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:36.465186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:36.465699 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:36.465319    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:25:46.465427 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:46.465388    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:25:52.339220 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.339193    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:25:52.431905 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.431842    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn2kx\" (UniqueName: \"kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx\") pod \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") "
Sep 18 20:25:52.431905 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.431875    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs\") pod \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\" (UID: \"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9\") "
Sep 18 20:25:52.434158 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.434125    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" (UID: "c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:52.434265 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.434208    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx" (OuterVolumeSpecName: "kube-api-access-nn2kx") pod "c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" (UID: "c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9"). InnerVolumeSpecName "kube-api-access-nn2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:52.532669 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.532639    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn2kx\" (UniqueName: \"kubernetes.io/projected/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-kube-api-access-nn2kx\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:52.532669 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.532666    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9-tls-certs\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:25:52.887996 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.887962    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerID="7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830" exitCode=0
Sep 18 20:25:52.888167 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.888010    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"
Sep 18 20:25:52.888167 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.888037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" event={"ID":"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9","Type":"ContainerDied","Data":"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"}
Sep 18 20:25:52.888167 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.888085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm" event={"ID":"c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9","Type":"ContainerDied","Data":"4b1cb48c3d820d1cd69206d2df4a96fede2e7b59be0c494b8ead4547a3f88c1e"}
Sep 18 20:25:52.888167 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.888108    2582 scope.go:117] "RemoveContainer" containerID="7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"
Sep 18 20:25:52.897139 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.897121    2582 scope.go:117] "RemoveContainer" containerID="7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"
Sep 18 20:25:52.897407 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:25:52.897386    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830\": container with ID starting with 7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830 not found: ID does not exist" containerID="7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"
Sep 18 20:25:52.897460 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.897415    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830"} err="failed to get container status \"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830\": rpc error: code = NotFound desc = could not find container \"7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830\": container with ID starting with 7a22ed6301cad886922a443429f7d0fe3ea93ec680c531b8a94055833ce2c830 not found: ID does not exist"
Sep 18 20:25:52.912366 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.912341    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:25:52.917390 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:52.917369    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zr4zm"]
Sep 18 20:25:53.895200 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:53.895165    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" path="/var/lib/kubelet/pods/c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9/volumes"
Sep 18 20:25:57.340008 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.339974    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xx24f/must-gather-f2vhk"]
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340252    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main"
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340263    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main"
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340284    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main"
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340289    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main"
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340342    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d6b638-24ff-402a-9eff-a8f5f7e0d2c9" containerName="main"
Sep 18 20:25:57.340389 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.340354    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="873e6ff5-c0b8-4962-9fb4-c31b3fafe629" containerName="main"
Sep 18 20:25:57.344753 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.344717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.349402 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.349385    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xx24f\"/\"default-dockercfg-dhlxm\""
Sep 18 20:25:57.350108 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.350088    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xx24f\"/\"openshift-service-ca.crt\""
Sep 18 20:25:57.350210 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.350165    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xx24f\"/\"kube-root-ca.crt\""
Sep 18 20:25:57.356853 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.356824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xx24f/must-gather-f2vhk"]
Sep 18 20:25:57.468516 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.468486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.468652 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.468521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rff\" (UniqueName: \"kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.569219 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.569185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.569363 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.569224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rff\" (UniqueName: \"kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.569517 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.569495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.586375 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.586352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rff\" (UniqueName: \"kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff\") pod \"must-gather-f2vhk\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") " pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.654081 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.654014    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:25:57.931552 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.931524    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xx24f/must-gather-f2vhk"]
Sep 18 20:25:57.933632 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:25:57.933606    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode79e4bf6_961d_437c_8953_9cfc3a463d23.slice/crio-6bcce85939a7c5ab15be7e8e4bd771a1f564c0f17ce3b130019faa2f9d43b58e WatchSource:0}: Error finding container 6bcce85939a7c5ab15be7e8e4bd771a1f564c0f17ce3b130019faa2f9d43b58e: Status 404 returned error can't find the container with id 6bcce85939a7c5ab15be7e8e4bd771a1f564c0f17ce3b130019faa2f9d43b58e
Sep 18 20:25:57.935249 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:57.935231    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:25:58.923848 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:25:58.923782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xx24f/must-gather-f2vhk" event={"ID":"e79e4bf6-961d-437c-8953-9cfc3a463d23","Type":"ContainerStarted","Data":"6bcce85939a7c5ab15be7e8e4bd771a1f564c0f17ce3b130019faa2f9d43b58e"}
Sep 18 20:26:04.959438 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:04.959395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xx24f/must-gather-f2vhk" event={"ID":"e79e4bf6-961d-437c-8953-9cfc3a463d23","Type":"ContainerStarted","Data":"5b802cf004f77b525cceb2a65dbd0ec4925530294fcadee4cd1f12a344f962c2"}
Sep 18 20:26:04.959913 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:04.959449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xx24f/must-gather-f2vhk" event={"ID":"e79e4bf6-961d-437c-8953-9cfc3a463d23","Type":"ContainerStarted","Data":"f0bfe2eb20685f34eab30759b92eaac0095d61b954d18e44887ce0a5ee723160"}
Sep 18 20:26:04.982870 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:04.982812    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xx24f/must-gather-f2vhk" podStartSLOduration=1.856577855 podStartE2EDuration="7.982795814s" totalImagesPullingTime="6.126217959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:04.980710988 +0000 UTC m=+4297.217837973" watchObservedRunningTime="2026-09-18 20:26:04.982795814 +0000 UTC m=+4297.219922799"
Sep 18 20:26:24.442371 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:24.442277    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5vb74_d2d24c61-ff61-41fb-ac59-28b59126824c/authorino/0.log"
Sep 18 20:26:24.495649 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:24.495616    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-kbnw5_d6a36572-2dfa-4cbf-92f0-29076a421408/manager/0.log"
Sep 18 20:26:24.623241 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:24.623183    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-5qmvp_9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5/manager/0.log"
Sep 18 20:26:26.070431 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.070396    2582 generic.go:356] "Generic (PLEG): container finished" podID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerID="f0bfe2eb20685f34eab30759b92eaac0095d61b954d18e44887ce0a5ee723160" exitCode=0
Sep 18 20:26:26.070887 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.070464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xx24f/must-gather-f2vhk" event={"ID":"e79e4bf6-961d-437c-8953-9cfc3a463d23","Type":"ContainerDied","Data":"f0bfe2eb20685f34eab30759b92eaac0095d61b954d18e44887ce0a5ee723160"}
Sep 18 20:26:26.070887 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.070788    2582 scope.go:117] "RemoveContainer" containerID="f0bfe2eb20685f34eab30759b92eaac0095d61b954d18e44887ce0a5ee723160"
Sep 18 20:26:26.294005 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.293976    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xx24f_must-gather-f2vhk_e79e4bf6-961d-437c-8953-9cfc3a463d23/gather/0.log"
Sep 18 20:26:26.957233 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.957146    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggjm/must-gather-m5rcr"]
Sep 18 20:26:26.960031 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.960008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:26.966129 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.965830    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"openshift-service-ca.crt\""
Sep 18 20:26:26.966129 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.965843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sggjm\"/\"default-dockercfg-gqtsx\""
Sep 18 20:26:26.970789 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.970769    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sggjm\"/\"kube-root-ca.crt\""
Sep 18 20:26:26.977432 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.977403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/must-gather-m5rcr"]
Sep 18 20:26:26.995710 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.995682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrcb\" (UniqueName: \"kubernetes.io/projected/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-kube-api-access-xnrcb\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:26.995830 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:26.995813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-must-gather-output\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.096824 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.096794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-must-gather-output\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.097173 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.096835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrcb\" (UniqueName: \"kubernetes.io/projected/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-kube-api-access-xnrcb\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.097173 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.097107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-must-gather-output\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.111235 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.111206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrcb\" (UniqueName: \"kubernetes.io/projected/94ab5c7a-1eeb-4ea4-8185-d18b6951fef4-kube-api-access-xnrcb\") pod \"must-gather-m5rcr\" (UID: \"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4\") " pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.268946 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.268902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/must-gather-m5rcr"
Sep 18 20:26:27.543329 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:27.543297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/must-gather-m5rcr"]
Sep 18 20:26:27.545247 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:26:27.545218    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ab5c7a_1eeb_4ea4_8185_d18b6951fef4.slice/crio-5c32ee18574f3d2717075828f6d2e83148c796f72e42f6740b07e8671e8acc9c WatchSource:0}: Error finding container 5c32ee18574f3d2717075828f6d2e83148c796f72e42f6740b07e8671e8acc9c: Status 404 returned error can't find the container with id 5c32ee18574f3d2717075828f6d2e83148c796f72e42f6740b07e8671e8acc9c
Sep 18 20:26:28.082194 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:28.082141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/must-gather-m5rcr" event={"ID":"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4","Type":"ContainerStarted","Data":"5c32ee18574f3d2717075828f6d2e83148c796f72e42f6740b07e8671e8acc9c"}
Sep 18 20:26:29.091886 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:29.091843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/must-gather-m5rcr" event={"ID":"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4","Type":"ContainerStarted","Data":"0d072d2b6a3553e6592a3d10d29c32576cbc758e7cc1621232467807e70811e9"}
Sep 18 20:26:29.092390 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:29.091901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/must-gather-m5rcr" event={"ID":"94ab5c7a-1eeb-4ea4-8185-d18b6951fef4","Type":"ContainerStarted","Data":"7ddefd8c8775c4e2ba56ec8c18143e08029417f03538432e5d372c0c4f7498e6"}
Sep 18 20:26:29.110977 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:29.110911    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggjm/must-gather-m5rcr" podStartSLOduration=2.451178887 podStartE2EDuration="3.110894144s" totalImagesPullingTime="659.715257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:29.10993348 +0000 UTC m=+4321.347060468" watchObservedRunningTime="2026-09-18 20:26:29.110894144 +0000 UTC m=+4321.348021130"
Sep 18 20:26:30.061213 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:30.061166    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-spxh5_2b36dced-2265-4d55-9ff6-e28b8102b04e/global-pull-secret-syncer/0.log"
Sep 18 20:26:30.221803 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:30.221769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-znpkc_e5c57212-561c-4172-9325-8d50d8b10155/konnectivity-agent/0.log"
Sep 18 20:26:30.306551 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:30.306520    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-75.ec2.internal_10e1249619055b5b051291a965f220aa/haproxy/0.log"
Sep 18 20:26:31.754122 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:31.754081    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-xx24f/must-gather-f2vhk"]
Sep 18 20:26:31.754645 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:31.754302    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-xx24f/must-gather-f2vhk" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="copy" containerID="cri-o://5b802cf004f77b525cceb2a65dbd0ec4925530294fcadee4cd1f12a344f962c2" gracePeriod=2
Sep 18 20:26:31.756467 ip-10-0-143-75 kubenswrapper[2582]: E0918 20:26:31.756421    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-f2vhk\" is forbidden: User \"system:node:ip-10-0-143-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xx24f\": no relationship found between node 'ip-10-0-143-75.ec2.internal' and this object" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:26:31.759247 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:31.759216    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-xx24f/must-gather-f2vhk"]
Sep 18 20:26:32.111810 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.111536    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xx24f_must-gather-f2vhk_e79e4bf6-961d-437c-8953-9cfc3a463d23/copy/0.log"
Sep 18 20:26:32.111977 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.111946    2582 generic.go:356] "Generic (PLEG): container finished" podID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerID="5b802cf004f77b525cceb2a65dbd0ec4925530294fcadee4cd1f12a344f962c2" exitCode=143
Sep 18 20:26:32.145750 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.144940    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xx24f_must-gather-f2vhk_e79e4bf6-961d-437c-8953-9cfc3a463d23/copy/0.log"
Sep 18 20:26:32.147100 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.147027    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:26:32.247313 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.247259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7rff\" (UniqueName: \"kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff\") pod \"e79e4bf6-961d-437c-8953-9cfc3a463d23\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") "
Sep 18 20:26:32.247527 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.247449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output\") pod \"e79e4bf6-961d-437c-8953-9cfc3a463d23\" (UID: \"e79e4bf6-961d-437c-8953-9cfc3a463d23\") "
Sep 18 20:26:32.255214 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.255173    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff" (OuterVolumeSpecName: "kube-api-access-n7rff") pod "e79e4bf6-961d-437c-8953-9cfc3a463d23" (UID: "e79e4bf6-961d-437c-8953-9cfc3a463d23"). InnerVolumeSpecName "kube-api-access-n7rff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:32.256810 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.256776    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e79e4bf6-961d-437c-8953-9cfc3a463d23" (UID: "e79e4bf6-961d-437c-8953-9cfc3a463d23"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:32.352237 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.352145    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rff\" (UniqueName: \"kubernetes.io/projected/e79e4bf6-961d-437c-8953-9cfc3a463d23-kube-api-access-n7rff\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:26:32.352237 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:32.352196    2582 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e79e4bf6-961d-437c-8953-9cfc3a463d23-must-gather-output\") on node \"ip-10-0-143-75.ec2.internal\" DevicePath \"\""
Sep 18 20:26:33.122661 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:33.122581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xx24f_must-gather-f2vhk_e79e4bf6-961d-437c-8953-9cfc3a463d23/copy/0.log"
Sep 18 20:26:33.123221 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:33.123115    2582 scope.go:117] "RemoveContainer" containerID="5b802cf004f77b525cceb2a65dbd0ec4925530294fcadee4cd1f12a344f962c2"
Sep 18 20:26:33.123315 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:33.123295    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xx24f/must-gather-f2vhk"
Sep 18 20:26:33.146062 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:33.145367    2582 scope.go:117] "RemoveContainer" containerID="f0bfe2eb20685f34eab30759b92eaac0095d61b954d18e44887ce0a5ee723160"
Sep 18 20:26:33.899764 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:33.899289    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" path="/var/lib/kubelet/pods/e79e4bf6-961d-437c-8953-9cfc3a463d23/volumes"
Sep 18 20:26:35.025813 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:35.025314    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5vb74_d2d24c61-ff61-41fb-ac59-28b59126824c/authorino/0.log"
Sep 18 20:26:35.200094 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:35.200067    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-kbnw5_d6a36572-2dfa-4cbf-92f0-29076a421408/manager/0.log"
Sep 18 20:26:35.379464 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:35.379377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-5qmvp_9b4d76ae-3bc5-4874-89f6-5cbeb064a8d5/manager/0.log"
Sep 18 20:26:37.779099 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:37.779054    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/node-exporter/0.log"
Sep 18 20:26:37.813645 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:37.813609    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/kube-rbac-proxy/0.log"
Sep 18 20:26:37.856193 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:37.856162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7g5s9_d9f386a8-eb6f-4edf-80cf-872a84b54f0c/init-textfile/0.log"
Sep 18 20:26:39.304669 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.304629    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"]
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305073    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="gather"
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305089    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="gather"
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305103    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="copy"
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305111    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="copy"
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305185    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="gather"
Sep 18 20:26:39.305369 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.305197    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79e4bf6-961d-437c-8953-9cfc3a463d23" containerName="copy"
Sep 18 20:26:39.309015 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.308991    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.320525 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.320491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"]
Sep 18 20:26:39.416782 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.416717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-lib-modules\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.416979 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.416829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-podres\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.416979 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.416896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-sys\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.416979 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.416931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-proc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.416979 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.416960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlsc\" (UniqueName: \"kubernetes.io/projected/7e75b994-5fb2-4043-a427-e4614d1df03b-kube-api-access-rjlsc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518110 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-sys\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-proc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlsc\" (UniqueName: \"kubernetes.io/projected/7e75b994-5fb2-4043-a427-e4614d1df03b-kube-api-access-rjlsc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-sys\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518319 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-proc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518544 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-lib-modules\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518544 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-lib-modules\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518544 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-podres\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.518544 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.518491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e75b994-5fb2-4043-a427-e4614d1df03b-podres\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.527235 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.527204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlsc\" (UniqueName: \"kubernetes.io/projected/7e75b994-5fb2-4043-a427-e4614d1df03b-kube-api-access-rjlsc\") pod \"perf-node-gather-daemonset-5h9wn\" (UID: \"7e75b994-5fb2-4043-a427-e4614d1df03b\") " pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.620998 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.620905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:39.913936 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:39.913903    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"]
Sep 18 20:26:39.918531 ip-10-0-143-75 kubenswrapper[2582]: W0918 20:26:39.916620    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e75b994_5fb2_4043_a427_e4614d1df03b.slice/crio-96c48242c3610f4a2bcd7f67a1d11e561b44e5e817bbfdd1b791dd4c40e03b29 WatchSource:0}: Error finding container 96c48242c3610f4a2bcd7f67a1d11e561b44e5e817bbfdd1b791dd4c40e03b29: Status 404 returned error can't find the container with id 96c48242c3610f4a2bcd7f67a1d11e561b44e5e817bbfdd1b791dd4c40e03b29
Sep 18 20:26:40.177832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.177720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn" event={"ID":"7e75b994-5fb2-4043-a427-e4614d1df03b","Type":"ContainerStarted","Data":"e8e5a23dbc50e548901704f83c238c74e273f44ac7cf29cd6dd84b747c8e4bf3"}
Sep 18 20:26:40.177832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.177803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn" event={"ID":"7e75b994-5fb2-4043-a427-e4614d1df03b","Type":"ContainerStarted","Data":"96c48242c3610f4a2bcd7f67a1d11e561b44e5e817bbfdd1b791dd4c40e03b29"}
Sep 18 20:26:40.177832 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.177824    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:40.202373 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.202322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn" podStartSLOduration=1.202307355 podStartE2EDuration="1.202307355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:40.202215629 +0000 UTC m=+4332.439342618" watchObservedRunningTime="2026-09-18 20:26:40.202307355 +0000 UTC m=+4332.439434372"
Sep 18 20:26:40.760458 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.760421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/2.log"
Sep 18 20:26:40.768089 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:40.768051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4slbl_f844729d-bf68-4b6a-9232-548cea591d4d/console-operator/3.log"
Sep 18 20:26:43.301713 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:43.301688    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jg4cv_39b83dbf-37c5-4f51-8eaa-9c313b54008f/dns/0.log"
Sep 18 20:26:43.333979 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:43.333953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jg4cv_39b83dbf-37c5-4f51-8eaa-9c313b54008f/kube-rbac-proxy/0.log"
Sep 18 20:26:43.404421 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:43.404393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kbjz4_9adc468a-d53a-4092-ab1f-73139aea1307/dns-node-resolver/0.log"
Sep 18 20:26:44.156470 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:44.156436    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwg8m_6cc65196-8ae5-48cb-8e49-9e8847db616a/node-ca/0.log"
Sep 18 20:26:46.197308 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:46.197272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sggjm/perf-node-gather-daemonset-5h9wn"
Sep 18 20:26:46.208224 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:46.208189    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6h4jg_9ee23005-dadc-47c4-ba3e-70e9eeadc2ec/serve-healthcheck-canary/0.log"
Sep 18 20:26:46.841575 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:46.841534    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tl8qx_70c89fe2-c674-42a9-870d-eb8b2d01de2b/insights-operator/0.log"
Sep 18 20:26:46.844181 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:46.844153    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tl8qx_70c89fe2-c674-42a9-870d-eb8b2d01de2b/insights-operator/1.log"
Sep 18 20:26:46.990423 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:46.990397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xz2d_0437e355-15cb-43cd-b240-cbf87cd7db60/kube-rbac-proxy/0.log"
Sep 18 20:26:47.070131 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:47.070100    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xz2d_0437e355-15cb-43cd-b240-cbf87cd7db60/exporter/0.log"
Sep 18 20:26:47.124499 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:47.124410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xz2d_0437e355-15cb-43cd-b240-cbf87cd7db60/extractor/0.log"
Sep 18 20:26:51.147386 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:51.147360    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85dd87cf5-rfcw8_8299c0f7-acdc-49ed-ad65-f332d0d899f4/manager/0.log"
Sep 18 20:26:53.715237 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:53.715203    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pphkx_4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80/s3-init/0.log"
Sep 18 20:26:53.749671 ip-10-0-143-75 kubenswrapper[2582]: I0918 20:26:53.749640    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pphkx_4874c7ab-0fbc-4a61-9c4d-cf8e5f777c80/download-hf-model/0.log"