Sep 17 18:39:41.402243 ip-10-0-136-146 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:39:44.851435 ip-10-0-136-146 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:44.851435 ip-10-0-136-146 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:39:44.851435 ip-10-0-136-146 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:44.851435 ip-10-0-136-146 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:39:44.851435 ip-10-0-136-146 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:44.852203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.851507    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:39:44.857347 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857328    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.857347 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857346    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857350    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857353    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857356    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857361    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857366    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857369    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857372    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857375    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857379    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857381    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857384    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857387    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857390    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857392    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857395    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857397    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857400    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857402    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.857418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857405    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857407    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857410    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857414    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857418    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857420    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857423    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857426    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857429    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857432    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857434    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857437    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857440    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857444    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857447    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857450    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857453    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857456    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857459    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.857869 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857462    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857465    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857467    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857470    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857473    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857475    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857478    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857481    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857483    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857486    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857489    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857492    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857494    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857497    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857499    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857503    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857506    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857509    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857512    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857514    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.858358 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857517    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857519    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857521    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857524    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857526    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857529    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857531    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857534    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857536    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857539    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857541    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857545    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857547    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857550    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857552    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857556    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857559    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857561    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857563    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857566    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.858843 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857569    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857571    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857573    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857576    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857579    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857581    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857583    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857587    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857590    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857593    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857595    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857598    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857600    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857603    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857605    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857608    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857611    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857613    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857616    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.857618    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.859333 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858397    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858404    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858409    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858413    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858416    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858419    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858422    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858425    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858428    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858431    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858434    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858436    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858439    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858442    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858444    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858447    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858449    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858452    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858455    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.859819 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858458    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858461    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858464    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858467    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858469    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858472    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858474    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858476    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858479    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858481    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858484    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858486    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858489    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858491    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858493    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858496    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858498    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858501    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858504    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858507    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.860345 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858510    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858512    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858516    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858519    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858522    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858525    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858528    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858531    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858533    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858536    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858539    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858543    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858546    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858549    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858552    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858554    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858557    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858559    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858562    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.860835 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858564    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858567    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858569    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858572    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858574    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858577    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858579    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858582    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858584    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858587    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858589    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858592    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858600    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858603    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858606    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858608    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858611    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858613    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858616    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858618    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.861308 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858621    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858624    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858627    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858630    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858633    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858635    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858641    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858643    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858646    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858648    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858651    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858653    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858656    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858658    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858661    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858664    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858666    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858668    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858671    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.861785 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.858674    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859139    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859146    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859149    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859153    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859156    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859159    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859162    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859165    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859168    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859171    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859173    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859176    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859179    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859182    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859184    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859187    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859190    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859193    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859195    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.862286 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859198    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859201    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859204    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859207    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859209    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859212    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859214    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859217    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859219    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859222    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859224    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859227    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859229    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859233    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859235    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859237    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859240    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859243    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859246    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859248    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.862761 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859251    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859254    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859256    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859259    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859261    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859264    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859268    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859272    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859275    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859278    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859280    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859283    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859285    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859288    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859291    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859293    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859295    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859298    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859301    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859303    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.863246 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859306    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859308    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859310    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859313    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859315    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859318    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859320    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859323    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859325    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859328    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859331    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859333    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859336    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859342    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859345    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859348    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859350    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859353    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859356    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859360    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.863735 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859364    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859368    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859371    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859373    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859377    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859381    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859384    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859387    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859390    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859392    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859395    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859398    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859400    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859403    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859405    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859408    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859410    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859413    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859416    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.864228 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.859418    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859490    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859525    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859533    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859539    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859544    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859548    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859553    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859561    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859565    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859568    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859571    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859575    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859580    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859583    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859586    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859589    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859592    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859595    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859600    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859603    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859606    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859609    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859613    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:39:44.864689 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859617    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859620    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859623    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859626    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859629    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859633    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859635    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859639    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859642    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859646    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859649    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859652    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859655    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859659    2561 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859664    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859669    2561 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859673    2561 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859675    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859678    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859681    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859685    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859689    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859692    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859695    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859698    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:39:44.865328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859701    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859705    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859708    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859711    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859713    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859716    2561 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859721    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859724    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859727    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859730    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859733    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859736    2561 flags.go:64] FLAG: --help="false"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859739    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-136-146.ec2.internal"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859742    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859745    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859748    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859751    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859754    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859757    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859760    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859763    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859767    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859771    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859774    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:39:44.865953 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859777    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859780    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859782    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859786    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859790    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859793    2561 flags.go:64] FLAG: --lock-file=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859796    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859798    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859801    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859807    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859810    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859812    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859815    2561 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859818    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859822    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859825    2561 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859828    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859832    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859835    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859839    2561 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859842    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859845    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859849    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859852    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859854    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:39:44.866837 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859858    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859861    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859867    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859870    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859875    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859879    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859881    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859887    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859904    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859908    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859911    2561 flags.go:64] FLAG: --port="10250"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859914    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859918    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e0c1a2274236e789"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859921    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859924    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859927    2561 flags.go:64] FLAG: --register-node="true"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859930    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859934    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859936    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859939    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859942    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859945    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859949    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859951    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859954    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:39:44.867945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859957    2561 flags.go:64] FLAG: --runonce="false"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859960    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859963    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859966    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859969    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859971    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859974    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859977    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859980    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859983    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859986    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859990    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859993    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859997    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.859999    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860002    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860007    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860010    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860013    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860019    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860022    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860025    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860028    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860031    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860034    2561 flags.go:64] FLAG: --v="2"
Sep 17 18:39:44.868874 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860038    2561 flags.go:64] FLAG: --version="false"
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860042    2561 flags.go:64] FLAG: --vmodule=""
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860046    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860049    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860227    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860232    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860235    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860238    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860241    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860244    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860247    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860249    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860252    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860255    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860257    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860260    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860262    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860265    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860269    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860272    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.869513 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860274    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860277    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860280    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860282    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860285    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860287    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860292    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860294    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860297    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860299    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860302    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860306    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860310    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860313    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860316    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860319    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860321    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860324    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860327    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.870087 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860330    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860332    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860335    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860338    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860340    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860343    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860345    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860348    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860350    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860353    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860356    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860360    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860363    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860365    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860368    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860372    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860374    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860377    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860379    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860383    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.870549 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860385    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860388    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860390    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860393    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860396    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860398    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860400    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860403    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860405    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860408    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860411    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860414    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860416    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860419    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860421    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860424    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860426    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860428    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860431    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860433    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.871067 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860437    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860440    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860443    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860447    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860449    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860452    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860454    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860457    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860460    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860463    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860465    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860469    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860472    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860474    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860477    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860479    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860482    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860485    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860487    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860489    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.871555 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860492    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860495    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860497    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.860500    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.860504    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.870316    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.870334    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.870356    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.870362    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870436    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870441    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870446    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870450    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870455    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.872160 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870458    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870461    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870464    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870467    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870470    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870472    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870475    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870478    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870480    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870483    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870486    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870489    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870491    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870494    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870497    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870502    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870504    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870507    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870510    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870513    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.872518 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870516    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870518    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870521    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870523    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870526    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870529    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870531    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870534    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870537    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870540    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870542    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870545    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870547    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870550    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870553    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870555    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870558    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870561    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870564    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870567    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.873507 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870569    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870572    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870575    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870577    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870580    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870583    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870585    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870588    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870591    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870594    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870596    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870599    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870602    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870604    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870607    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870610    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870613    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870616    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870618    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870621    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.874063 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870624    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870626    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870629    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870631    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870634    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870637    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870639    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870642    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870644    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870647    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870649    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870652    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870655    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870658    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870661    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870664    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870667    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870669    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870672    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870674    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.874556 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870677    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870679    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870682    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870684    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870687    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870689    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870692    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870694    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870697    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870700    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870703    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870705    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870709    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870713    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.875071 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.870718    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870884    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870905    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870909    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870911    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870914    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870917    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870920    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870923    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870926    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870928    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870933    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870936    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870940    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870943    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870946    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870949    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870951    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870954    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870957    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:44.875418 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870959    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870962    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870965    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870967    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870970    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870972    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870975    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870978    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870980    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870983    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870985    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870988    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870991    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870994    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.870997    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871001    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871003    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871006    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871009    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:44.875913 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871011    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871013    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871016    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871018    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871022    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871026    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871029    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871032    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871035    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871038    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871040    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871043    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871046    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871048    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871051    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871054    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871057    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871059    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871062    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871064    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:44.876386 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871067    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871070    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871073    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871075    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871078    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871081    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871084    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871086    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871088    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871091    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871093    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871096    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871098    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871100    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871103    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871105    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871108    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871110    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871113    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871115    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:44.876871 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871118    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871121    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871123    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871126    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871129    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871131    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871134    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871136    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871139    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871141    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871143    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871146    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871149    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871152    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871154    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871157    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871159    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871162    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871165    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:44.877373 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871167    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:44.871170    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.871175    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.871303    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.873537    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.874559    2561 server.go:1019] "Starting client certificate rotation"
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.874664    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.874704    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:44.877824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.877082    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:44.878435 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.878410    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:44.882642 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.882623    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:39:44.884582 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.884569    2561 log.go:25] "Validated CRI v1 image API"
Sep 17 18:39:44.890281 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.886679    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:39:44.890281 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.889590    2561 fs.go:135] Filesystem UUIDs: map[6f2d4ca5-1c7f-4116-8980-21e07c60175a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f7f8933d-6385-4164-9b13-674527d7e0a3:/dev/nvme0n1p3]
Sep 17 18:39:44.890281 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.889608    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:39:44.894743 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.894623    2561 manager.go:216] Machine: {Timestamp:2026-09-17 18:39:44.894129233 +0000 UTC m=+2.035555439 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101443 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223a940be045883aa92b882d76793d SystemUUID:ec223a94-0be0-4588-3aa9-2b882d76793d BootID:296ef7c1-cdf7-4ca2-b5c8-49a587578a17 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:14:0e:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:14:0e:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:29:6b:01:f5:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:39:44.894743 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.894731    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:39:44.894885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.894814    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:39:44.896424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.896396    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:39:44.896563 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.896427    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:39:44.897765 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.897754    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:39:44.897806 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.897767    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:39:44.897806 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.897776    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:44.897806 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.897787    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:44.898932 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.898922    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:39:44.901872 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.901860    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:44.902007 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.901997    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:39:44.902061 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902041    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:39:44.902061 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902052    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:39:44.902128 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902066    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:39:44.902128 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902076    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:39:44.902128 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902084    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:39:44.902461 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902435    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:44.902461 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.902453    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:44.904328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.904307    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:39:44.906875 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.906855    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:39:44.906979 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.906889    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:39:44.908160 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908147    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908163    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908169    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908174    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908183    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908188    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908202    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:39:44.908208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908207    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908214    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908220    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908229    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908241    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908270    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:39:44.908415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908275    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:39:44.908630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.908447    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:44.912193 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.912179    2561 server.go:1268] "Started kubelet"
Sep 17 18:39:44.912315 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.912253    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:39:44.912315 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.912280    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:39:44.912436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.912318    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:39:44.912519 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.912466    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:39:44.913072 ip-10-0-136-146 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:39:44.913526 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.913322    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:39:44.913814 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.913738    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:39:44.917946 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.917924    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:39:44.918435 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.918414    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:39:44.919709 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.919695    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:39:44.919783 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.919771    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:44.919840 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.919812    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.920249    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.920258    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.920340    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.920351    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.920457    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:39:44.920453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.920463    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:39:44.945045 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.945014    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:39:44.945184 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.945126    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:39:44.945312 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.945261    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:39:44.945394 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.945315    2561 factory.go:55] Registering systemd factory
Sep 17 18:39:44.945394 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.945325    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:39:44.945538 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.945506    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b766"
Sep 17 18:39:44.946014 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.945999    2561 factory.go:153] Registering CRI-O factory
Sep 17 18:39:44.946252 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.946227    2561 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:39:44.946347 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.946340    2561 factory.go:103] Registering Raw factory
Sep 17 18:39:44.946427 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.946420    2561 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:39:44.946868 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.946851    2561 manager.go:349] Starting recovery of all containers
Sep 17 18:39:44.947431 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.947408    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b766"
Sep 17 18:39:44.952502 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:44.951470    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-146.ec2.internal.18d62fbba5b9af29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-146.ec2.internal,UID:ip-10-0-136-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-146.ec2.internal,},FirstTimestamp:2026-09-17 18:39:44.912150313 +0000 UTC m=+2.053576519,LastTimestamp:2026-09-17 18:39:44.912150313 +0000 UTC m=+2.053576519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-146.ec2.internal,}"
Sep 17 18:39:44.958206 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.958190    2561 manager.go:354] Recovery completed
Sep 17 18:39:44.962040 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.962020    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:44.963639 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.963623    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:44.963709 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.963659    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:44.963709 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.963671    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:44.964185 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964170    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:39:44.964185 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964181    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:44.964334 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964182    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:39:44.964334 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964238    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:44.964869 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964855    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:44.964937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964882    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:44.964937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.964912    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:44.965313 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.965300    2561 policy_none.go:49] "None policy: Start"
Sep 17 18:39:44.965373 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.965320    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:39:44.965373 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.965335    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:39:44.966231 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:44.966218    2561 policy_none.go:47] "Start"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001047    2561 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.001081    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001092    2561 server.go:87] "Starting device plugin registration server"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001372    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001395    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001513    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001603    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.001611    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.002129    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:39:45.014251 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.002189    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.079022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.078981    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:39:45.080367 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.080350    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:39:45.080452 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.080413    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:39:45.080452 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.080446    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:39:45.080550 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.080495    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:39:45.083037 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.083014    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:45.102039 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.101981    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:45.103009 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.102993    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:45.103089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.103025    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:45.103089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.103039    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:45.103089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.103073    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.109457 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.109443    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.109520 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.109466    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-146.ec2.internal\": node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.130915 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.130870    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.181505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.181459    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"]
Sep 17 18:39:45.183294 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.181857    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:45.184299 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.184280    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:45.184414 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.184316    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:45.184414 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.184329    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:45.185585 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.185571    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:45.185751 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.185735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.185836 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.185813    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:45.186363 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186345    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:45.186436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186351    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:45.186436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186401    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:45.186436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186412    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:45.186436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186383    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:45.186559 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.186445    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:45.187423 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.187406    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.187521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.187431    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:45.188088 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.188067    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:45.188166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.188106    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:45.188166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.188123    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:45.216700 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.216677    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-146.ec2.internal\" not found" node="ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.221214 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.221197    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-146.ec2.internal\" not found" node="ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.222236 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.222219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.222296 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.222243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.222296 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.222261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/056c7b5706208129dfc0377a36f97f3c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-146.ec2.internal\" (UID: \"056c7b5706208129dfc0377a36f97f3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.230973 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.230955    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.322748 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.322748 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.322966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/056c7b5706208129dfc0377a36f97f3c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-146.ec2.internal\" (UID: \"056c7b5706208129dfc0377a36f97f3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.322966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/056c7b5706208129dfc0377a36f97f3c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-146.ec2.internal\" (UID: \"056c7b5706208129dfc0377a36f97f3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.322966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.322966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.322826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a71f409c16266b0fe2f5303cfe9d8f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal\" (UID: \"6a71f409c16266b0fe2f5303cfe9d8f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.331771 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.331751    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.431930 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.431807    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.520046 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.520019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.525257 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.525118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:45.532765 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.532741    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.633244 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.633204    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.733710 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.733642    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.834150 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.834121    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.874316 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.874286    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:39:45.874704 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.874438    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:39:45.901213 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.901189    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:45.934779 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:45.934747    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:45.951958 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.951908    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:34:44 +0000 UTC" deadline="2028-03-11 23:38:33.253739301 +0000 UTC"
Sep 17 18:39:45.952075 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:45.951969    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12988h58m47.301790904s"
Sep 17 18:39:46.034862 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:46.034832    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-146.ec2.internal\" not found"
Sep 17 18:39:46.037716 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.037688    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:46.120557 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.120514    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:46.128717 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.128698    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:46.130354 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.130340    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal"
Sep 17 18:39:46.139528 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:46.139499    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a71f409c16266b0fe2f5303cfe9d8f2.slice/crio-217531163d09cff37e7b981d10e7141e12ea7f72467d3a88755ddc5f2bfd7e59 WatchSource:0}: Error finding container 217531163d09cff37e7b981d10e7141e12ea7f72467d3a88755ddc5f2bfd7e59: Status 404 returned error can't find the container with id 217531163d09cff37e7b981d10e7141e12ea7f72467d3a88755ddc5f2bfd7e59
Sep 17 18:39:46.139965 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:46.139948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056c7b5706208129dfc0377a36f97f3c.slice/crio-3e3216a68a9da420057db3eda02a591b31d50d4aa2f4fdc2dfd42943a30e6d8c WatchSource:0}: Error finding container 3e3216a68a9da420057db3eda02a591b31d50d4aa2f4fdc2dfd42943a30e6d8c: Status 404 returned error can't find the container with id 3e3216a68a9da420057db3eda02a591b31d50d4aa2f4fdc2dfd42943a30e6d8c
Sep 17 18:39:46.143368 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.143353    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:46.145012 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.144994    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:39:46.391248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.391161    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:46.903987 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.903948    2561 apiserver.go:52] "Watching apiserver"
Sep 17 18:39:46.912027 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.911989    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:39:46.912439 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.912405    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rzhx6","kube-system/konnectivity-agent-v6jfk","kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal","openshift-multus/multus-additional-cni-plugins-qkmb7","openshift-multus/multus-qqkfr","openshift-ovn-kubernetes/ovnkube-node-22lq4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv","openshift-cluster-node-tuning-operator/tuned-bdcg2","openshift-dns/node-resolver-vqtgn","openshift-image-registry/node-ca-p7tld","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal","openshift-multus/network-metrics-daemon-b9cf4","openshift-network-diagnostics/network-check-target-lpcg6","openshift-network-operator/iptables-alerter-f76hg"]
Sep 17 18:39:46.914012 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.913989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.915640 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.915374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:46.915640 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:46.915454    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:46.916550 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.916520    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-889xn\""
Sep 17 18:39:46.916658 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.916593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.916760 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.916744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.916816 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.916781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:39:46.916993 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.916977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:46.918583 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.918560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.920476 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.920458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.920578 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.920511    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:39:46.920923 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.920886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tsk8\""
Sep 17 18:39:46.921031 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.921003    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.921168 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.920887    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:39:46.921323 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.921239    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.921571 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.921556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thh2h\""
Sep 17 18:39:46.921851 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.921825    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.922211 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.922196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.922697 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.922681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.923180 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.923158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:46.923619 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.923600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:39:46.923785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.923766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:46.923968 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:46.923873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:46.924927 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.924889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:39:46.925247 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925232    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:39:46.925410 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925392    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:39:46.925473 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925417    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqxpc\""
Sep 17 18:39:46.925537 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925523    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.925697 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925685    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qb2mk\""
Sep 17 18:39:46.925830 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925817    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:39:46.925968 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.925950    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:46.926802 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.926327    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:39:46.926802 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.926503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:39:46.926802 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.926507    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zppv\""
Sep 17 18:39:46.926802 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.926700    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:46.927408 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.927390    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.928146 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.927587    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.928146 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.927594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.928146 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.927823    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:46.929171 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.929150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:46.930416 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.930524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd2216fe-b76c-4e0c-aed3-0533173eacb1-hosts-file\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:46.930524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f9z8\" (UniqueName: \"kubernetes.io/projected/8e79dc1c-f159-4707-8051-2b4b315d1048-kube-api-access-8f9z8\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.930524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-systemd-units\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.930524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-var-lib-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.930524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-bin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:46.930576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-kubelet\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-etc-kubernetes\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-ovn\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.930767 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-env-overrides\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cnibin\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-cni-binary-copy\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-systemd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd2216fe-b76c-4e0c-aed3-0533173eacb1-tmp-dir\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nqs\" (UniqueName: \"kubernetes.io/projected/dd2216fe-b76c-4e0c-aed3-0533173eacb1-kube-api-access-k4nqs\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252ht\" (UniqueName: \"kubernetes.io/projected/86309611-2ddf-49d7-8806-05ec5b3928ca-kube-api-access-252ht\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-netns\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.930952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-node-log\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63335224-a809-468b-9c26-86a7c3ba6a77-agent-certs\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktqz\" (UniqueName: \"kubernetes.io/projected/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-kube-api-access-jktqz\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-multus\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-dbus\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-cnibin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-socket-dir-parent\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-iptables-alerter-script\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-hostroot\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-slash\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-etc-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f856\" (UniqueName: \"kubernetes.io/projected/26c17490-9136-4f0e-8e2f-ade74d14bdd6-kube-api-access-8f856\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63335224-a809-468b-9c26-86a7c3ba6a77-konnectivity-ca\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-system-cni-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-device-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-netd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.931799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-multus-certs\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-kubelet\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-daemon-config\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbq6\" (UniqueName: \"kubernetes.io/projected/15e4ce2c-6fb3-4ad9-a073-f50607348e91-kube-api-access-8lbq6\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-kubelet-config\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-config\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-host-slash\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.931976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-k8s-cni-cncf-io\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-netns\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-log-socket\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-bin\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-system-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-conf-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:46.932444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovn-node-metrics-cert\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.933059 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.932200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-script-lib\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:46.933059 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.933044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jj4dg\""
Sep 17 18:39:46.933251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.933231    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.933999    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhnps\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934043    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934079    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934325    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6j4b8\""
Sep 17 18:39:46.934441 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.934334    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:46.938234 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.938073    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:46.963002 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.962956    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sgwk7"
Sep 17 18:39:46.971671 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:46.971317    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sgwk7"
Sep 17 18:39:47.021461 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.021428    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:39:47.032972 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.032939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-iptables-alerter-script\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.032983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-hostroot\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-slash\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-etc-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f856\" (UniqueName: \"kubernetes.io/projected/26c17490-9136-4f0e-8e2f-ade74d14bdd6-kube-api-access-8f856\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63335224-a809-468b-9c26-86a7c3ba6a77-konnectivity-ca\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-conf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-system-cni-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.033166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-device-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-netd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-multus-certs\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-kubelet\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5nw\" (UniqueName: \"kubernetes.io/projected/64e79258-76fa-4c91-922a-a43748ad79ac-kube-api-access-8c5nw\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-var-lib-kubelet\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-daemon-config\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64e79258-76fa-4c91-922a-a43748ad79ac-host\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-modprobe-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbq6\" (UniqueName: \"kubernetes.io/projected/15e4ce2c-6fb3-4ad9-a073-f50607348e91-kube-api-access-8lbq6\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.033601 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-kubelet-config\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-config\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-host-slash\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-iptables-alerter-script\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-kubernetes\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-sys\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-host\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-hostroot\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-registration-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-k8s-cni-cncf-io\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-netns\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-k8s-cni-cncf-io\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-log-socket\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-slash\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-bin\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.034402 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-system-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-etc-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.033974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-conf-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovn-node-metrics-cert\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-script-lib\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-systemd\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-run-netns\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-etc-tuned\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-log-socket\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-bin\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd2216fe-b76c-4e0c-aed3-0533173eacb1-hosts-file\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-system-cni-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f9z8\" (UniqueName: \"kubernetes.io/projected/8e79dc1c-f159-4707-8051-2b4b315d1048-kube-api-access-8f9z8\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-systemd-units\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-var-lib-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-lib-modules\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzkf\" (UniqueName: \"kubernetes.io/projected/11287709-b89f-473f-9208-22f2bc367404-kube-api-access-hdzkf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-bin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-tmp\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-kubelet\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-etc-kubernetes\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-ovn\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63335224-a809-468b-9c26-86a7c3ba6a77-konnectivity-ca\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-env-overrides\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-system-cni-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysconfig\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-run\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.035962 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cnibin\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-cni-netd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-cni-binary-copy\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-systemd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd2216fe-b76c-4e0c-aed3-0533173eacb1-tmp-dir\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nqs\" (UniqueName: \"kubernetes.io/projected/dd2216fe-b76c-4e0c-aed3-0533173eacb1-kube-api-access-k4nqs\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252ht\" (UniqueName: \"kubernetes.io/projected/86309611-2ddf-49d7-8806-05ec5b3928ca-kube-api-access-252ht\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-netns\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-node-log\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64e79258-76fa-4c91-922a-a43748ad79ac-serviceca\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-conf-dir\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63335224-a809-468b-9c26-86a7c3ba6a77-agent-certs\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktqz\" (UniqueName: \"kubernetes.io/projected/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-kube-api-access-jktqz\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.035070    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-multus\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.036694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vth\" (UniqueName: \"kubernetes.io/projected/05ff87bb-d370-4f09-921b-a00049948b21-kube-api-access-z7vth\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.035147    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.535124736 +0000 UTC m=+4.676550954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-multus-certs\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.035941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.034699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-device-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.036243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.036242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-sys-fs\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.036361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-daemon-config\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.036434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-node-log\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-host-kubelet\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd2216fe-b76c-4e0c-aed3-0533173eacb1-tmp-dir\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-etc-kubernetes\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-kubelet-config\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.037453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cnibin\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-ovn\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e79dc1c-f159-4707-8051-2b4b315d1048-cni-binary-copy\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-host-slash\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e4ce2c-6fb3-4ad9-a073-f50607348e91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-config\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-dbus\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-run-netns\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-cnibin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037793    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-socket-dir-parent\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.037969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-multus-socket-dir-parent\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038008    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-kubelet\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b8b0027-07c0-4dde-b7af-6418d620ef89-dbus\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-env-overrides\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-cnibin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-bin\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86309611-2ddf-49d7-8806-05ec5b3928ca-socket-dir\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd2216fe-b76c-4e0c-aed3-0533173eacb1-hosts-file\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-systemd-units\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-var-lib-openvswitch\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e79dc1c-f159-4707-8051-2b4b315d1048-host-var-lib-cni-multus\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26c17490-9136-4f0e-8e2f-ade74d14bdd6-run-systemd\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.038946 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.038604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovnkube-script-lib\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.043025 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.042991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26c17490-9136-4f0e-8e2f-ade74d14bdd6-ovn-node-metrics-cert\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.043187 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.043107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63335224-a809-468b-9c26-86a7c3ba6a77-agent-certs\") pod \"konnectivity-agent-v6jfk\" (UID: \"63335224-a809-468b-9c26-86a7c3ba6a77\") " pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:47.047959 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.047933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-252ht\" (UniqueName: \"kubernetes.io/projected/86309611-2ddf-49d7-8806-05ec5b3928ca-kube-api-access-252ht\") pod \"aws-ebs-csi-driver-node-8b6wv\" (UID: \"86309611-2ddf-49d7-8806-05ec5b3928ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.048971 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.048941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nqs\" (UniqueName: \"kubernetes.io/projected/dd2216fe-b76c-4e0c-aed3-0533173eacb1-kube-api-access-k4nqs\") pod \"node-resolver-vqtgn\" (UID: \"dd2216fe-b76c-4e0c-aed3-0533173eacb1\") " pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.049078 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.048974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbq6\" (UniqueName: \"kubernetes.io/projected/15e4ce2c-6fb3-4ad9-a073-f50607348e91-kube-api-access-8lbq6\") pod \"multus-additional-cni-plugins-qkmb7\" (UID: \"15e4ce2c-6fb3-4ad9-a073-f50607348e91\") " pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.053291 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.052827    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:47.053291 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.052855    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:47.053291 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.052875    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.053291 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.052965    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.552943837 +0000 UTC m=+4.694370040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.056248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.056217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f9z8\" (UniqueName: \"kubernetes.io/projected/8e79dc1c-f159-4707-8051-2b4b315d1048-kube-api-access-8f9z8\") pod \"multus-qqkfr\" (UID: \"8e79dc1c-f159-4707-8051-2b4b315d1048\") " pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.057292 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.057265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f856\" (UniqueName: \"kubernetes.io/projected/26c17490-9136-4f0e-8e2f-ade74d14bdd6-kube-api-access-8f856\") pod \"ovnkube-node-22lq4\" (UID: \"26c17490-9136-4f0e-8e2f-ade74d14bdd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.058581 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.058537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktqz\" (UniqueName: \"kubernetes.io/projected/1d45ff25-8a81-4e6d-899c-ee7dafc5e66a-kube-api-access-jktqz\") pod \"iptables-alerter-f76hg\" (UID: \"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a\") " pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.086280 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.086214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal" event={"ID":"056c7b5706208129dfc0377a36f97f3c","Type":"ContainerStarted","Data":"3e3216a68a9da420057db3eda02a591b31d50d4aa2f4fdc2dfd42943a30e6d8c"}
Sep 17 18:39:47.087515 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.087482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal" event={"ID":"6a71f409c16266b0fe2f5303cfe9d8f2","Type":"ContainerStarted","Data":"217531163d09cff37e7b981d10e7141e12ea7f72467d3a88755ddc5f2bfd7e59"}
Sep 17 18:39:47.139173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-conf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5nw\" (UniqueName: \"kubernetes.io/projected/64e79258-76fa-4c91-922a-a43748ad79ac-kube-api-access-8c5nw\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-var-lib-kubelet\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64e79258-76fa-4c91-922a-a43748ad79ac-host\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-modprobe-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-kubernetes\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-sys\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-host\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139401 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-systemd\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-var-lib-kubelet\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-etc-tuned\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-modprobe-d\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysctl-conf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-lib-modules\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-kubernetes\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzkf\" (UniqueName: \"kubernetes.io/projected/11287709-b89f-473f-9208-22f2bc367404-kube-api-access-hdzkf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-host\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-tmp\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-systemd\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64e79258-76fa-4c91-922a-a43748ad79ac-host\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysconfig\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-run\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64e79258-76fa-4c91-922a-a43748ad79ac-serviceca\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.139656    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:47.139866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vth\" (UniqueName: \"kubernetes.io/projected/05ff87bb-d370-4f09-921b-a00049948b21-kube-api-access-z7vth\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.139706    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.63969052 +0000 UTC m=+4.781116710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-lib-modules\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-run\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.139943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-etc-sysconfig\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.140049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11287709-b89f-473f-9208-22f2bc367404-sys\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.140853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.140259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64e79258-76fa-4c91-922a-a43748ad79ac-serviceca\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.142220 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.142196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-tmp\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.142327 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.142226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11287709-b89f-473f-9208-22f2bc367404-etc-tuned\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.152311 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.152287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5nw\" (UniqueName: \"kubernetes.io/projected/64e79258-76fa-4c91-922a-a43748ad79ac-kube-api-access-8c5nw\") pod \"node-ca-p7tld\" (UID: \"64e79258-76fa-4c91-922a-a43748ad79ac\") " pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.152416 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.152286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vth\" (UniqueName: \"kubernetes.io/projected/05ff87bb-d370-4f09-921b-a00049948b21-kube-api-access-z7vth\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.154213 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.154195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzkf\" (UniqueName: \"kubernetes.io/projected/11287709-b89f-473f-9208-22f2bc367404-kube-api-access-hdzkf\") pod \"tuned-bdcg2\" (UID: \"11287709-b89f-473f-9208-22f2bc367404\") " pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.231088 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.230989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv"
Sep 17 18:39:47.240392 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.240359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f76hg"
Sep 17 18:39:47.249066 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.249038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkmb7"
Sep 17 18:39:47.255797 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.255777    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqkfr"
Sep 17 18:39:47.264423 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.264402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:39:47.270968 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.270949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vqtgn"
Sep 17 18:39:47.274864 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.274845    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:47.280562 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.280546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:39:47.286150 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.286131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p7tld"
Sep 17 18:39:47.292885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.292867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bdcg2"
Sep 17 18:39:47.542232 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.542142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:47.542396 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.542274    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:47.542396 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.542337    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:48.542317601 +0000 UTC m=+5.683743810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:47.643097 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.643052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:47.643269 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.643111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:47.643269 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643252    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:47.643346 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643316    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:48.643295103 +0000 UTC m=+5.784721312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:47.643764 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643746    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:47.643865 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643772    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:47.643865 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643786    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.643865 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:47.643832    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:48.643816404 +0000 UTC m=+5.785242605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:47.974086 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.973993    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:46 +0000 UTC" deadline="2028-02-21 11:53:54.421974027 +0000 UTC"
Sep 17 18:39:47.974086 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:47.974030    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12521h14m6.447947776s"
Sep 17 18:39:48.031560 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.031526    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86309611_2ddf_49d7_8806_05ec5b3928ca.slice/crio-096b5c68e84474dd6ff413679a08c4fe21a83b38c0532948d0bbb62b5d05089a WatchSource:0}: Error finding container 096b5c68e84474dd6ff413679a08c4fe21a83b38c0532948d0bbb62b5d05089a: Status 404 returned error can't find the container with id 096b5c68e84474dd6ff413679a08c4fe21a83b38c0532948d0bbb62b5d05089a
Sep 17 18:39:48.053763 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.053392    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2216fe_b76c_4e0c_aed3_0533173eacb1.slice/crio-f115f6bb57bd3b0683ac03706be9edc44cf8708902fd446f71a934189c422335 WatchSource:0}: Error finding container f115f6bb57bd3b0683ac03706be9edc44cf8708902fd446f71a934189c422335: Status 404 returned error can't find the container with id f115f6bb57bd3b0683ac03706be9edc44cf8708902fd446f71a934189c422335
Sep 17 18:39:48.056362 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.056339    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d45ff25_8a81_4e6d_899c_ee7dafc5e66a.slice/crio-47ae313b3da446176a02792bc001d0eab4d6115fb2efbb606fa84af6d6be800e WatchSource:0}: Error finding container 47ae313b3da446176a02792bc001d0eab4d6115fb2efbb606fa84af6d6be800e: Status 404 returned error can't find the container with id 47ae313b3da446176a02792bc001d0eab4d6115fb2efbb606fa84af6d6be800e
Sep 17 18:39:48.057104 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.057048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c17490_9136_4f0e_8e2f_ade74d14bdd6.slice/crio-40e328146f0216d435a3e8e4e84bee6acfc1b49d78b6309979764bf824f83965 WatchSource:0}: Error finding container 40e328146f0216d435a3e8e4e84bee6acfc1b49d78b6309979764bf824f83965: Status 404 returned error can't find the container with id 40e328146f0216d435a3e8e4e84bee6acfc1b49d78b6309979764bf824f83965
Sep 17 18:39:48.061044 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.061020    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e4ce2c_6fb3_4ad9_a073_f50607348e91.slice/crio-9a083cc8b9cbe9081901718abf423e83cf65da1447c52855ac0d19c78d5a2dd9 WatchSource:0}: Error finding container 9a083cc8b9cbe9081901718abf423e83cf65da1447c52855ac0d19c78d5a2dd9: Status 404 returned error can't find the container with id 9a083cc8b9cbe9081901718abf423e83cf65da1447c52855ac0d19c78d5a2dd9
Sep 17 18:39:48.061948 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.061929    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e79dc1c_f159_4707_8051_2b4b315d1048.slice/crio-dad2a82c0c6ffc1ed085010757edbde5472ab73d0efe9dfddd896d5e362b6064 WatchSource:0}: Error finding container dad2a82c0c6ffc1ed085010757edbde5472ab73d0efe9dfddd896d5e362b6064: Status 404 returned error can't find the container with id dad2a82c0c6ffc1ed085010757edbde5472ab73d0efe9dfddd896d5e362b6064
Sep 17 18:39:48.062753 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.062730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e79258_76fa_4c91_922a_a43748ad79ac.slice/crio-11dea68ca4789fec3f2090ac9d29a21783d99b2d3d775df704317b668dcccd11 WatchSource:0}: Error finding container 11dea68ca4789fec3f2090ac9d29a21783d99b2d3d775df704317b668dcccd11: Status 404 returned error can't find the container with id 11dea68ca4789fec3f2090ac9d29a21783d99b2d3d775df704317b668dcccd11
Sep 17 18:39:48.064380 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:39:48.064112    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11287709_b89f_473f_9208_22f2bc367404.slice/crio-d8965e3274bcf523acb8a427baeebe5578ca42feea3dc042c1ce07d177f77be2 WatchSource:0}: Error finding container d8965e3274bcf523acb8a427baeebe5578ca42feea3dc042c1ce07d177f77be2: Status 404 returned error can't find the container with id d8965e3274bcf523acb8a427baeebe5578ca42feea3dc042c1ce07d177f77be2
Sep 17 18:39:48.080695 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.080668    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:48.080695 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.080689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:48.080864 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.080797    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:48.080981 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.080956    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:48.096939 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.096884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqtgn" event={"ID":"dd2216fe-b76c-4e0c-aed3-0533173eacb1","Type":"ContainerStarted","Data":"f115f6bb57bd3b0683ac03706be9edc44cf8708902fd446f71a934189c422335"}
Sep 17 18:39:48.099463 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.099437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv" event={"ID":"86309611-2ddf-49d7-8806-05ec5b3928ca","Type":"ContainerStarted","Data":"096b5c68e84474dd6ff413679a08c4fe21a83b38c0532948d0bbb62b5d05089a"}
Sep 17 18:39:48.100614 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.100586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6jfk" event={"ID":"63335224-a809-468b-9c26-86a7c3ba6a77","Type":"ContainerStarted","Data":"af354dd4b143d33147e470f5f678166edba228d7715ff45fc25fe9cec0eeeaa5"}
Sep 17 18:39:48.101950 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.101925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bdcg2" event={"ID":"11287709-b89f-473f-9208-22f2bc367404","Type":"ContainerStarted","Data":"d8965e3274bcf523acb8a427baeebe5578ca42feea3dc042c1ce07d177f77be2"}
Sep 17 18:39:48.103043 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.103017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqkfr" event={"ID":"8e79dc1c-f159-4707-8051-2b4b315d1048","Type":"ContainerStarted","Data":"dad2a82c0c6ffc1ed085010757edbde5472ab73d0efe9dfddd896d5e362b6064"}
Sep 17 18:39:48.104052 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.104026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerStarted","Data":"9a083cc8b9cbe9081901718abf423e83cf65da1447c52855ac0d19c78d5a2dd9"}
Sep 17 18:39:48.105089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.105064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"40e328146f0216d435a3e8e4e84bee6acfc1b49d78b6309979764bf824f83965"}
Sep 17 18:39:48.108035 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.108012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f76hg" event={"ID":"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a","Type":"ContainerStarted","Data":"47ae313b3da446176a02792bc001d0eab4d6115fb2efbb606fa84af6d6be800e"}
Sep 17 18:39:48.111580 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.111554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p7tld" event={"ID":"64e79258-76fa-4c91-922a-a43748ad79ac","Type":"ContainerStarted","Data":"11dea68ca4789fec3f2090ac9d29a21783d99b2d3d775df704317b668dcccd11"}
Sep 17 18:39:48.551710 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.551399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:48.551865 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.551543    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:48.551865 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.551796    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.551774874 +0000 UTC m=+7.693201078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:48.652963 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.652922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:48.653132 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.652978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:48.653132 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653103    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:48.653248 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.653146561 +0000 UTC m=+7.794572774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:48.653606 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653585    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:48.653699 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653611    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:48.653699 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653625    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:48.653699 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:48.653675    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:50.653657366 +0000 UTC m=+7.795083562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:48.975314 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.975181    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:46 +0000 UTC" deadline="2028-03-05 23:25:01.831683124 +0000 UTC"
Sep 17 18:39:48.975314 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:48.975249    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12844h45m12.856438683s"
Sep 17 18:39:49.083177 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:49.083142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:49.083342 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:49.083277    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:49.119763 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:49.119725    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a71f409c16266b0fe2f5303cfe9d8f2" containerID="2e809fde65db5b3f42fdd4bfb7e498103e8006bb7136dfe6037fc610c30980cc" exitCode=0
Sep 17 18:39:49.119960 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:49.119832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal" event={"ID":"6a71f409c16266b0fe2f5303cfe9d8f2","Type":"ContainerDied","Data":"2e809fde65db5b3f42fdd4bfb7e498103e8006bb7136dfe6037fc610c30980cc"}
Sep 17 18:39:49.144727 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:49.144688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal" event={"ID":"056c7b5706208129dfc0377a36f97f3c","Type":"ContainerStarted","Data":"832d0a8fcfb92c7b3d1cfba8db6e45ea7d2d1c84e65a38e35c52a90352b18738"}
Sep 17 18:39:50.081087 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.081044    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:50.081605 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.081209    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:50.081692 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.081655    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:50.081776 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.081750    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:50.182024 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.181959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal" event={"ID":"6a71f409c16266b0fe2f5303cfe9d8f2","Type":"ContainerStarted","Data":"d7720e56a2a6d0b96ad84e3acfd69fef3016f2459ebd9207475e95af01ba4f24"}
Sep 17 18:39:50.205037 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.204959    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-146.ec2.internal" podStartSLOduration=4.204936551 podStartE2EDuration="4.204936551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:50.203917574 +0000 UTC m=+7.345343785" watchObservedRunningTime="2026-09-17 18:39:50.204936551 +0000 UTC m=+7.346362761"
Sep 17 18:39:50.205366 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.205336    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-146.ec2.internal" podStartSLOduration=4.20532629 podStartE2EDuration="4.20532629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:49.168431513 +0000 UTC m=+6.309857725" watchObservedRunningTime="2026-09-17 18:39:50.20532629 +0000 UTC m=+7.346752504"
Sep 17 18:39:50.566596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.565863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:50.566596 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.566118    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:50.566596 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.566183    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.566163759 +0000 UTC m=+11.707589954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.667225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:50.667273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.667435    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.667525    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.667479798 +0000 UTC m=+11.808905990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.667975    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.667993    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.668007    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:50.668088 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:50.668051    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.668036765 +0000 UTC m=+11.809462958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:51.083757 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:51.083721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:51.084211 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:51.083860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:52.081017 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:52.080979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:52.081202 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:52.081103    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:52.081270 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:52.081204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:52.081364 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:52.081328    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:53.081344 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:53.081280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:53.081781 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:53.081422    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:54.081211 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:54.081170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:54.081383 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:54.081213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:54.081383 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.081316    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:54.081738 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.081717    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:54.601333 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:54.601287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:54.601537 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.601439    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:54.601537 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.601497    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:02.601483235 +0000 UTC m=+19.742909424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:54.702522 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:54.702467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:54.702522 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:54.702528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:54.702776 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.702669    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:54.702776 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.702731    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:02.702715971 +0000 UTC m=+19.844142173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:54.703178 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.703159    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:54.703295 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.703184    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:54.703295 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.703195    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:54.703295 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:54.703238    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:02.703223412 +0000 UTC m=+19.844649615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:55.084488 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:55.084408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:55.084928 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:55.084534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:56.081599 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:56.081565    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:56.081863 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:56.081780    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:56.082217 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:56.082198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:56.082318 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:56.082294    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:57.082955 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:57.081275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:57.083433 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:57.083039    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:39:58.081600 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:58.081553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:39:58.081600 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:58.081582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:39:58.081831 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:58.081716    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:39:58.081885 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:58.081836    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:39:59.081501 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:39:59.081458    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:39:59.081963 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:39:59.081606    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:00.080839 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:00.080800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:00.081052 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:00.080953    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:00.081052 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:00.081014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:00.081134 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:00.081106    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:01.080735 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:01.080692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:01.081230 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:01.080838    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:02.081201 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:02.081112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:02.081201 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:02.081157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:02.081649 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.081265    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:02.081649 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.081409    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:02.662315 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:02.662274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:02.662908 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.662867    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:02.663012 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.662965    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:18.662943666 +0000 UTC m=+35.804369869 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:02.763203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:02.763164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:02.763387 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:02.763212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:02.763387 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763340    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:02.763387 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763365    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:40:02.763560 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763390    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:40:02.763560 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763404    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:02.763560 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763411    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:18.763391495 +0000 UTC m=+35.904817700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:02.763560 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:02.763456    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:18.763439915 +0000 UTC m=+35.904866105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:03.081094 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:03.081008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:03.081257 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:03.081226    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:04.081649 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:04.081598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:04.081649 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:04.081637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:04.082205 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:04.081733    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:04.082205 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:04.082185    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:05.084490 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:05.084462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:05.084956 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:05.084579    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:06.080939 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.080913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:06.081049 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.080954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:06.081114 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:06.081051    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:06.081247 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:06.081216    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:06.219137 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.219059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6jfk" event={"ID":"63335224-a809-468b-9c26-86a7c3ba6a77","Type":"ContainerStarted","Data":"5a157c2b142519b8ef54902f82bae995d8620df4c7a72e4841d988cae5c73fb0"}
Sep 17 18:40:06.225296 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.224725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bdcg2" event={"ID":"11287709-b89f-473f-9208-22f2bc367404","Type":"ContainerStarted","Data":"8fc170b59549eed07212a16544e0de8cb04afedf9e1e6c9f16e0622201f9b254"}
Sep 17 18:40:06.230594 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.229654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerStarted","Data":"1ab62746cce49a7ced1e7ea726545a59e3215a78d71beefb13cbfa7711cb2133"}
Sep 17 18:40:06.232306 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.232090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"45da36fa59966101dc7ea4057fde6b3588951a21a4e0785bf575f77561f0dccb"}
Sep 17 18:40:06.237360 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.237256    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v6jfk" podStartSLOduration=3.392568523 podStartE2EDuration="21.23723697s" totalImagesPullingTime="17.844668447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:06.236831855 +0000 UTC m=+23.378258068" watchObservedRunningTime="2026-09-17 18:40:06.23723697 +0000 UTC m=+23.378663185"
Sep 17 18:40:06.280231 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:06.280037    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bdcg2" podStartSLOduration=3.391528903 podStartE2EDuration="21.280019559s" totalImagesPullingTime="17.888490656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:06.279594213 +0000 UTC m=+23.421020424" watchObservedRunningTime="2026-09-17 18:40:06.280019559 +0000 UTC m=+23.421445771"
Sep 17 18:40:07.081058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.080804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:07.081267 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:07.081185    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:07.235723 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.235628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqkfr" event={"ID":"8e79dc1c-f159-4707-8051-2b4b315d1048","Type":"ContainerStarted","Data":"c36282b642a64837f95114eb12bf69f02f1471944899eab6d09fb6df59b929c3"}
Sep 17 18:40:07.237179 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.237143    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="1ab62746cce49a7ced1e7ea726545a59e3215a78d71beefb13cbfa7711cb2133" exitCode=0
Sep 17 18:40:07.237289 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.237216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"1ab62746cce49a7ced1e7ea726545a59e3215a78d71beefb13cbfa7711cb2133"}
Sep 17 18:40:07.240501 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.240479    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:40:07.240855 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.240833    2561 generic.go:356] "Generic (PLEG): container finished" podID="26c17490-9136-4f0e-8e2f-ade74d14bdd6" containerID="082203b41828e21947752149f36a0acc8de26efdacc741fde1420d7bea9ae8f2" exitCode=1
Sep 17 18:40:07.240975 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.240918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"c722a5f166d0d6e95f9243ea670bbc48380eb6e2d93d0c6e1ea9a5e5587d1cf3"}
Sep 17 18:40:07.241034 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.241006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"5b76918677fae797da60c48a8a9b0b01292a3cbd32db6b89cc0e62c6dc5e7bea"}
Sep 17 18:40:07.241087 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.241057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"1dfaa69661a5a551495b97ea51470261c95b4ed01fce33664b6b88b97bbc7ee6"}
Sep 17 18:40:07.241139 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.241108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"07586f7137d78d3c16a7c7f9ff96f6797b9399f6118ad680f7ef2690609143f6"}
Sep 17 18:40:07.241197 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.241157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerDied","Data":"082203b41828e21947752149f36a0acc8de26efdacc741fde1420d7bea9ae8f2"}
Sep 17 18:40:07.242414 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.242384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p7tld" event={"ID":"64e79258-76fa-4c91-922a-a43748ad79ac","Type":"ContainerStarted","Data":"0b15768b27e4fd84a887a4380ea857b7529f182a9bdade30136aabd34b50ac1d"}
Sep 17 18:40:07.246824 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.246791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vqtgn" event={"ID":"dd2216fe-b76c-4e0c-aed3-0533173eacb1","Type":"ContainerStarted","Data":"19ccb0c2421f3dd7a1a8d6ac9e595cd77d8d380e0702f57ac1fad392d56b3084"}
Sep 17 18:40:07.248329 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.248305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv" event={"ID":"86309611-2ddf-49d7-8806-05ec5b3928ca","Type":"ContainerStarted","Data":"ae86eec2ad8393a286087cb727a86e84d178cbf9d20903dd24591a86094c11e9"}
Sep 17 18:40:07.254220 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.254170    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qqkfr" podStartSLOduration=4.315180172 podStartE2EDuration="22.254158318s" totalImagesPullingTime="17.938978146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:07.253836668 +0000 UTC m=+24.395262880" watchObservedRunningTime="2026-09-17 18:40:07.254158318 +0000 UTC m=+24.395584529"
Sep 17 18:40:07.295686 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.295636    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p7tld" podStartSLOduration=4.372169871 podStartE2EDuration="22.295617859s" totalImagesPullingTime="17.923447988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:07.271842246 +0000 UTC m=+24.413268458" watchObservedRunningTime="2026-09-17 18:40:07.295617859 +0000 UTC m=+24.437044071"
Sep 17 18:40:07.629488 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:07.629463    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:40:08.010595 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.010429    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:40:07.629481974Z","UUID":"fe640e55-8618-41fb-a032-d433dc35b566","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:40:08.012219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.012197    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:40:08.012340 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.012234    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:40:08.081233 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.081199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:08.081429 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:08.081353    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:08.081429 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.081404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:08.081547 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:08.081512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:08.254134 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.254045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv" event={"ID":"86309611-2ddf-49d7-8806-05ec5b3928ca","Type":"ContainerStarted","Data":"443304b378ccd67eacfa732e95d98beca07b56c6d83205d54244bb6b27ebdb25"}
Sep 17 18:40:08.255909 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.255869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f76hg" event={"ID":"1d45ff25-8a81-4e6d-899c-ee7dafc5e66a","Type":"ContainerStarted","Data":"ac7022571e14461a868492571728e788687193de1501df18a3f86db217afb2d9"}
Sep 17 18:40:08.277023 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.276916    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vqtgn" podStartSLOduration=5.417863057 podStartE2EDuration="23.276874038s" totalImagesPullingTime="17.859010981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:07.317782774 +0000 UTC m=+24.459208980" watchObservedRunningTime="2026-09-17 18:40:08.276874038 +0000 UTC m=+25.418300250"
Sep 17 18:40:08.277252 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:08.277222    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f76hg" podStartSLOduration=5.343540015 podStartE2EDuration="23.277209652s" totalImagesPullingTime="17.933669637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.277058177 +0000 UTC m=+25.418484390" watchObservedRunningTime="2026-09-17 18:40:08.277209652 +0000 UTC m=+25.418635864"
Sep 17 18:40:09.080827 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:09.080731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:09.081023 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:09.080850    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:09.263915 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:09.263839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv" event={"ID":"86309611-2ddf-49d7-8806-05ec5b3928ca","Type":"ContainerStarted","Data":"72223c32f6e0028d2dc2ecf1d66170cf6e26a42526e835cef03a4926c5da8e51"}
Sep 17 18:40:09.269024 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:09.268983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:40:09.269589 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:09.269555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"db5f0758697513f3b506ec1b2bcb88ff11fecea2d012961bd9dfdbba63fc9d4b"}
Sep 17 18:40:09.285306 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:09.285248    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b6wv" podStartSLOduration=4.117569039 podStartE2EDuration="24.285232822s" totalImagesPullingTime="20.167663783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:09.285017833 +0000 UTC m=+26.426444046" watchObservedRunningTime="2026-09-17 18:40:09.285232822 +0000 UTC m=+26.426659074"
Sep 17 18:40:10.081788 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:10.081588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:10.082028 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:10.081588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:10.082028 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:10.081921    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:10.082163 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:10.082043    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:10.743350 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:10.743311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:40:10.744036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:10.744011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:40:11.081158 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:11.081125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:11.081340 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:11.081258    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:11.273978 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:11.273938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:40:11.274546 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:11.274525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6jfk"
Sep 17 18:40:12.081462 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.081426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:12.082094 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.081475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:12.082094 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:12.081563    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:12.082094 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:12.081711    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:12.280763 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.280570    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:40:12.281251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.281044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"ebd6344849f3908bd934f9f1354a61a00ec913705ded6e7065fff838b4dc1244"}
Sep 17 18:40:12.281414 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.281386    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:12.281480 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.281467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:12.281534 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.281491    2561 scope.go:117] "RemoveContainer" containerID="082203b41828e21947752149f36a0acc8de26efdacc741fde1420d7bea9ae8f2"
Sep 17 18:40:12.281584 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.281500    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:12.299298 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.299275    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:12.299429 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:12.299375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:13.081677 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.081482    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:13.082347 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:13.081751    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:13.288552 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.288519    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqtgn_dd2216fe-b76c-4e0c-aed3-0533173eacb1/dns-node-resolver/0.log"
Sep 17 18:40:13.288745 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.288614    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:40:13.289087 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.289060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" event={"ID":"26c17490-9136-4f0e-8e2f-ade74d14bdd6","Type":"ContainerStarted","Data":"95e3e49aa2de666fa6bf2678ab156be1fbda28f16057ad1229b04ee828f6bdc8"}
Sep 17 18:40:13.291068 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.291042    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="eaf444524c484a189040329661253373e1238cdd40f05ef236bf9ea747e90817" exitCode=0
Sep 17 18:40:13.291174 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.291126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"eaf444524c484a189040329661253373e1238cdd40f05ef236bf9ea747e90817"}
Sep 17 18:40:13.334427 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:13.334302    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4" podStartSLOduration=10.405120623 podStartE2EDuration="28.334284489s" totalImagesPullingTime="17.929163866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:13.329927503 +0000 UTC m=+30.471353714" watchObservedRunningTime="2026-09-17 18:40:13.334284489 +0000 UTC m=+30.475710755"
Sep 17 18:40:14.081316 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.081282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:14.081481 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:14.081393    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:14.081481 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.081444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:14.081583 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:14.081554    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:14.245475 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.245447    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p7tld_64e79258-76fa-4c91-922a-a43748ad79ac/node-ca/0.log"
Sep 17 18:40:14.597251 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.597207    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpcg6"]
Sep 17 18:40:14.597405 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.597341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:14.597472 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:14.597446    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:14.598009 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.597986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzhx6"]
Sep 17 18:40:14.598107 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.598070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:14.598164 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:14.598149    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:14.600769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.600746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b9cf4"]
Sep 17 18:40:14.600876 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:14.600827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:14.600963 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:14.600932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:15.298256 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:15.298175    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="40efab0e1da9a9f6a7ba9d3c45bd43ef933397ae92c8676ce13a0db6ee3228c4" exitCode=0
Sep 17 18:40:15.298256 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:15.298245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"40efab0e1da9a9f6a7ba9d3c45bd43ef933397ae92c8676ce13a0db6ee3228c4"}
Sep 17 18:40:16.080742 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:16.080709    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:16.080914 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:16.080776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:16.080914 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:16.080887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:16.081023 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:16.080926    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:16.081023 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:16.081003    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:16.081108 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:16.081078    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:17.305909 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:17.305854    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="c59abd0957d37d632e29923f68d1e66f88b1a135e838895ba541483cb990f2c1" exitCode=0
Sep 17 18:40:17.306397 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:17.305949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"c59abd0957d37d632e29923f68d1e66f88b1a135e838895ba541483cb990f2c1"}
Sep 17 18:40:18.081022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.080986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:18.081022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.081021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:18.081336 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.081021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:18.081336 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.081124    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:18.081336 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.081220    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:18.081336 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.081286    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:18.692878 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.692830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:18.693345 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.693006    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:18.693345 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.693072    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret podName:1b8b0027-07c0-4dde-b7af-6418d620ef89 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:50.693056718 +0000 UTC m=+67.834482908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret") pod "global-pull-secret-syncer-rzhx6" (UID: "1b8b0027-07c0-4dde-b7af-6418d620ef89") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:40:18.793655 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.793615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:18.793879 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:18.793666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:18.793879 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793825    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:18.793879 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793830    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:40:18.793879 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793864    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:40:18.793879 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793880    2561 projected.go:196] Error preparing data for projected volume kube-api-access-6dnd8 for pod openshift-network-diagnostics/network-check-target-lpcg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:18.794141 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793917    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs podName:05ff87bb-d370-4f09-921b-a00049948b21 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:50.793880484 +0000 UTC m=+67.935306674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs") pod "network-metrics-daemon-b9cf4" (UID: "05ff87bb-d370-4f09-921b-a00049948b21") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:40:18.794141 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:18.793967    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8 podName:39a299bf-7624-4e5d-a444-4d7580f04259 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:50.793954557 +0000 UTC m=+67.935380759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6dnd8" (UniqueName: "kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8") pod "network-check-target-lpcg6" (UID: "39a299bf-7624-4e5d-a444-4d7580f04259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:40:20.081301 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:20.081259    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:20.081763 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:20.081307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:20.081763 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:20.081307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:20.081763 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:20.081393    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:20.081763 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:20.081493    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:20.081763 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:20.081596    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:22.080655 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:22.080627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:22.080655 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:22.080654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:22.081323 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:22.080630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:22.081323 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:22.080734    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:22.081323 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:22.080835    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:22.081323 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:22.080929    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:22.323536 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:22.323497    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="ca35a5bd16db7cab02bf21699de7bbdf1c813fe4d5fc65352aa99aeb9d675a64" exitCode=0
Sep 17 18:40:22.323752 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:22.323577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"ca35a5bd16db7cab02bf21699de7bbdf1c813fe4d5fc65352aa99aeb9d675a64"}
Sep 17 18:40:23.329833 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:23.329798    2561 generic.go:356] "Generic (PLEG): container finished" podID="15e4ce2c-6fb3-4ad9-a073-f50607348e91" containerID="517c9676aa2e0801063f43fd1da5f2e37e7846355a5fe86edc3362730d5a9068" exitCode=0
Sep 17 18:40:23.330412 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:23.329878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerDied","Data":"517c9676aa2e0801063f43fd1da5f2e37e7846355a5fe86edc3362730d5a9068"}
Sep 17 18:40:24.081507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:24.081461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:24.081734 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:24.081476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:24.081734 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:24.081599    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:24.081734 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:24.081476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:24.081734 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:24.081678    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:24.081863 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:24.081733    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:24.336950 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:24.336873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" event={"ID":"15e4ce2c-6fb3-4ad9-a073-f50607348e91","Type":"ContainerStarted","Data":"3214223bfb62a3be39f109b8550ea6cc69f910b3610f005032732000471bebf8"}
Sep 17 18:40:24.364596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:24.364538    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qkmb7" podStartSLOduration=10.175122282 podStartE2EDuration="39.36452048s" totalImagesPullingTime="29.189398198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:24.363103127 +0000 UTC m=+41.504529339" watchObservedRunningTime="2026-09-17 18:40:24.36452048 +0000 UTC m=+41.505946724"
Sep 17 18:40:26.081396 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:26.081336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:26.081973 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:26.081336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:26.081973 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:26.081474    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:26.081973 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:26.081336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:26.081973 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:26.081553    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:26.081973 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:26.081638    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:28.080877 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:28.080799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:28.080877 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:28.080857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:28.080877 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:28.080829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:28.081434 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:28.080988    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:28.081434 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:28.081102    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:28.081434 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:28.081170    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:30.080770 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:30.080736    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:30.081173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:30.080814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:30.081173 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:30.080925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:30.081173 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:30.080940    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:30.081173 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:30.080999    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:30.081173 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:30.081052    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:32.081520 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:32.081469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:32.081520 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:32.081518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:32.082012 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:32.081487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:32.082012 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:32.081623    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:32.082012 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:32.081746    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:32.082012 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:32.081836    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:34.081305 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.081260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:34.081305 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.081309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:34.081967 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.081364    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:34.081967 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:34.081376    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lpcg6" podUID="39a299bf-7624-4e5d-a444-4d7580f04259"
Sep 17 18:40:34.081967 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:34.081483    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9cf4" podUID="05ff87bb-d370-4f09-921b-a00049948b21"
Sep 17 18:40:34.081967 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:34.081552    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzhx6" podUID="1b8b0027-07c0-4dde-b7af-6418d620ef89"
Sep 17 18:40:34.739318 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.739287    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-146.ec2.internal" event="NodeReady"
Sep 17 18:40:34.739483 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.739423    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:40:34.977758 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.977665    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74bc5b974-xsm24"]
Sep 17 18:40:34.998952 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:34.998885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.004908 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.004868    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2z5j4"]
Sep 17 18:40:35.007305 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.007280    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:40:35.010340 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.010316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:40:35.013161 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.013135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:40:35.013375 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.013346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-spfxb\""
Sep 17 18:40:35.024968 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.024931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.027023 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.026992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bc5b974-xsm24"]
Sep 17 18:40:35.033249 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.033225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:40:35.034659 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.034634    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kzdh4\""
Sep 17 18:40:35.034775 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.034659    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:40:35.034775 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.034658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:40:35.034775 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.034684    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:40:35.037083 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.037066    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:40:35.054232 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.054195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2z5j4"]
Sep 17 18:40:35.074706 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.074675    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-npxf9"]
Sep 17 18:40:35.098269 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.098159    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.101002 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.100972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:40:35.102059 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.102031    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:40:35.102233 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.102134    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l4x2p\""
Sep 17 18:40:35.102233 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.102225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:40:35.102605 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.102582    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q88zg"]
Sep 17 18:40:35.114016 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.113986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-certificates\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w624t\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-kube-api-access-w624t\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c947bcc9-94d2-4709-a322-8547e2a9ccb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-image-registry-private-configuration\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsf4\" (UniqueName: \"kubernetes.io/projected/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-api-access-hpsf4\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-trusted-ca\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114153 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e1253ab-f0d2-407d-b53e-2ab940f3317a-ca-trust-extracted\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114369 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-installation-pull-secrets\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114369 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.114369 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c947bcc9-94d2-4709-a322-8547e2a9ccb2-crio-socket\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.114369 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c947bcc9-94d2-4709-a322-8547e2a9ccb2-data-volume\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.114509 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-tls\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114509 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-bound-sa-token\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.114712 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114692    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-npxf9"]
Sep 17 18:40:35.114757 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114726    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q88zg"]
Sep 17 18:40:35.114810 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.114800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.121180 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.121159    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj7h2\""
Sep 17 18:40:35.121325 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.121198    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:40:35.121325 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.121219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:40:35.215771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-image-registry-private-configuration\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.215771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsf4\" (UniqueName: \"kubernetes.io/projected/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-api-access-hpsf4\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a78cc5a-39bd-4867-921b-dee7019942e6-config-volume\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a78cc5a-39bd-4867-921b-dee7019942e6-metrics-tls\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-trusted-ca\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a78cc5a-39bd-4867-921b-dee7019942e6-tmp-dir\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e1253ab-f0d2-407d-b53e-2ab940f3317a-ca-trust-extracted\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.215998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-installation-pull-secrets\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-cert\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.216060 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c947bcc9-94d2-4709-a322-8547e2a9ccb2-crio-socket\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c947bcc9-94d2-4709-a322-8547e2a9ccb2-data-volume\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnkh\" (UniqueName: \"kubernetes.io/projected/0a78cc5a-39bd-4867-921b-dee7019942e6-kube-api-access-kqnkh\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-tls\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-bound-sa-token\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c947bcc9-94d2-4709-a322-8547e2a9ccb2-crio-socket\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhg7b\" (UniqueName: \"kubernetes.io/projected/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-kube-api-access-vhg7b\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-certificates\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w624t\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-kube-api-access-w624t\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c947bcc9-94d2-4709-a322-8547e2a9ccb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.216887 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e1253ab-f0d2-407d-b53e-2ab940f3317a-ca-trust-extracted\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.216887 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.216478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c947bcc9-94d2-4709-a322-8547e2a9ccb2-data-volume\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.217089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.217059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.217166 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.217147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-trusted-ca\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.217432 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.217238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-certificates\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.220722 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.220685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c947bcc9-94d2-4709-a322-8547e2a9ccb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.220835 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.220741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-registry-tls\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.220835 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.220759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-installation-pull-secrets\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.220835 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.220741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e1253ab-f0d2-407d-b53e-2ab940f3317a-image-registry-private-configuration\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.234936 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.234835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w624t\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-kube-api-access-w624t\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.235134 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.235045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsf4\" (UniqueName: \"kubernetes.io/projected/c947bcc9-94d2-4709-a322-8547e2a9ccb2-kube-api-access-hpsf4\") pod \"insights-runtime-extractor-2z5j4\" (UID: \"c947bcc9-94d2-4709-a322-8547e2a9ccb2\") " pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.236202 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.236178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e1253ab-f0d2-407d-b53e-2ab940f3317a-bound-sa-token\") pod \"image-registry-74bc5b974-xsm24\" (UID: \"5e1253ab-f0d2-407d-b53e-2ab940f3317a\") " pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.309522 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.309478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:35.316707 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a78cc5a-39bd-4867-921b-dee7019942e6-config-volume\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.316799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a78cc5a-39bd-4867-921b-dee7019942e6-metrics-tls\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.316799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a78cc5a-39bd-4867-921b-dee7019942e6-tmp-dir\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.316799 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-cert\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.316956 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnkh\" (UniqueName: \"kubernetes.io/projected/0a78cc5a-39bd-4867-921b-dee7019942e6-kube-api-access-kqnkh\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.316956 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.316856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhg7b\" (UniqueName: \"kubernetes.io/projected/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-kube-api-access-vhg7b\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.317170 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.317146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a78cc5a-39bd-4867-921b-dee7019942e6-tmp-dir\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.317393 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.317364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a78cc5a-39bd-4867-921b-dee7019942e6-config-volume\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.319435 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.319404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a78cc5a-39bd-4867-921b-dee7019942e6-metrics-tls\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.319544 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.319527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-cert\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.330913 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.330870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhg7b\" (UniqueName: \"kubernetes.io/projected/54482567-ff8b-4b5b-a824-e6dbf27fd9dd-kube-api-access-vhg7b\") pod \"ingress-canary-npxf9\" (UID: \"54482567-ff8b-4b5b-a824-e6dbf27fd9dd\") " pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.331973 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.331958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnkh\" (UniqueName: \"kubernetes.io/projected/0a78cc5a-39bd-4867-921b-dee7019942e6-kube-api-access-kqnkh\") pod \"dns-default-q88zg\" (UID: \"0a78cc5a-39bd-4867-921b-dee7019942e6\") " pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.335908 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.335875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2z5j4"
Sep 17 18:40:35.412698 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.412664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-npxf9"
Sep 17 18:40:35.423057 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.422997    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:35.681168 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.681140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2z5j4"]
Sep 17 18:40:35.684014 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:35.683981    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc947bcc9_94d2_4709_a322_8547e2a9ccb2.slice/crio-f5cfbc94bde8920c7a3e0a1386e34f4a2a1009a5e08c26507bb13fe8d97052b9 WatchSource:0}: Error finding container f5cfbc94bde8920c7a3e0a1386e34f4a2a1009a5e08c26507bb13fe8d97052b9: Status 404 returned error can't find the container with id f5cfbc94bde8920c7a3e0a1386e34f4a2a1009a5e08c26507bb13fe8d97052b9
Sep 17 18:40:35.686183 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.686149    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bc5b974-xsm24"]
Sep 17 18:40:35.691103 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:35.691061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1253ab_f0d2_407d_b53e_2ab940f3317a.slice/crio-039530ff7f03498fd3f488afc6c1ae35388b1694fa8c7f919c7ec849856b6d97 WatchSource:0}: Error finding container 039530ff7f03498fd3f488afc6c1ae35388b1694fa8c7f919c7ec849856b6d97: Status 404 returned error can't find the container with id 039530ff7f03498fd3f488afc6c1ae35388b1694fa8c7f919c7ec849856b6d97
Sep 17 18:40:35.726735 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.726552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-npxf9"]
Sep 17 18:40:35.730320 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:35.730275    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54482567_ff8b_4b5b_a824_e6dbf27fd9dd.slice/crio-96d6bf1a8b03dbebf73901bca88876936396ac352f23fc0fb61d2f466d5b2273 WatchSource:0}: Error finding container 96d6bf1a8b03dbebf73901bca88876936396ac352f23fc0fb61d2f466d5b2273: Status 404 returned error can't find the container with id 96d6bf1a8b03dbebf73901bca88876936396ac352f23fc0fb61d2f466d5b2273
Sep 17 18:40:35.749099 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:35.749065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q88zg"]
Sep 17 18:40:35.763179 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:35.763147    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a78cc5a_39bd_4867_921b_dee7019942e6.slice/crio-d65ba6c832dc7cfd25c371a8b3790c1d487735303764bc56cb2ff72f85b9215a WatchSource:0}: Error finding container d65ba6c832dc7cfd25c371a8b3790c1d487735303764bc56cb2ff72f85b9215a: Status 404 returned error can't find the container with id d65ba6c832dc7cfd25c371a8b3790c1d487735303764bc56cb2ff72f85b9215a
Sep 17 18:40:36.081057 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.081023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:36.081252 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.081024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:36.081252 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.081031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:36.084484 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:40:36.084644 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:40:36.084644 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:40:36.084644 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:40:36.084961 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:40:36.085031 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.084980    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:40:36.369730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.369633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z5j4" event={"ID":"c947bcc9-94d2-4709-a322-8547e2a9ccb2","Type":"ContainerStarted","Data":"99398dec90bc31c6bec1be49c63335f01fcd15054b4134b79808c60c8694686d"}
Sep 17 18:40:36.369730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.369712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z5j4" event={"ID":"c947bcc9-94d2-4709-a322-8547e2a9ccb2","Type":"ContainerStarted","Data":"f5cfbc94bde8920c7a3e0a1386e34f4a2a1009a5e08c26507bb13fe8d97052b9"}
Sep 17 18:40:36.371643 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.371614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bc5b974-xsm24" event={"ID":"5e1253ab-f0d2-407d-b53e-2ab940f3317a","Type":"ContainerStarted","Data":"3988dd4867e354d4ae6543ca517797acfeb110453a16c678046cbc7f2a8dfb20"}
Sep 17 18:40:36.371787 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.371682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bc5b974-xsm24" event={"ID":"5e1253ab-f0d2-407d-b53e-2ab940f3317a","Type":"ContainerStarted","Data":"039530ff7f03498fd3f488afc6c1ae35388b1694fa8c7f919c7ec849856b6d97"}
Sep 17 18:40:36.371787 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.371710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:36.373085 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.373062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q88zg" event={"ID":"0a78cc5a-39bd-4867-921b-dee7019942e6","Type":"ContainerStarted","Data":"d65ba6c832dc7cfd25c371a8b3790c1d487735303764bc56cb2ff72f85b9215a"}
Sep 17 18:40:36.374502 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.374467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-npxf9" event={"ID":"54482567-ff8b-4b5b-a824-e6dbf27fd9dd","Type":"ContainerStarted","Data":"96d6bf1a8b03dbebf73901bca88876936396ac352f23fc0fb61d2f466d5b2273"}
Sep 17 18:40:36.398101 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:36.398047    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74bc5b974-xsm24" podStartSLOduration=2.398025285 podStartE2EDuration="2.398025285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:36.396719274 +0000 UTC m=+53.538145488" watchObservedRunningTime="2026-09-17 18:40:36.398025285 +0000 UTC m=+53.539451496"
Sep 17 18:40:37.380751 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:37.380239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z5j4" event={"ID":"c947bcc9-94d2-4709-a322-8547e2a9ccb2","Type":"ContainerStarted","Data":"74c5b1af5ffd86c2eb352a6150cb9f5b48b663b136e0de32f71e149c78b2a852"}
Sep 17 18:40:40.391278 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.391238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-npxf9" event={"ID":"54482567-ff8b-4b5b-a824-e6dbf27fd9dd","Type":"ContainerStarted","Data":"8612fea584e3a5fb8774a32e7b59cb0188f875d55423236cf3bedd08a5a0907e"}
Sep 17 18:40:40.393416 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.393390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2z5j4" event={"ID":"c947bcc9-94d2-4709-a322-8547e2a9ccb2","Type":"ContainerStarted","Data":"531a682da0fa17086bc055e9d88bd0a207f4c94e1acb3f3ab8bc6eb081602a58"}
Sep 17 18:40:40.395083 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.395061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q88zg" event={"ID":"0a78cc5a-39bd-4867-921b-dee7019942e6","Type":"ContainerStarted","Data":"284987a499e43f376e9730c29bd387a62592bd85e30262555742d85b24409c47"}
Sep 17 18:40:40.395180 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.395103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q88zg" event={"ID":"0a78cc5a-39bd-4867-921b-dee7019942e6","Type":"ContainerStarted","Data":"1cc38f4459061b5b680b97a3be6bc265bc047e5c786052f4d2c60506548e8bc7"}
Sep 17 18:40:40.395219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.395192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:40.409126 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.409080    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-npxf9" podStartSLOduration=1.7254666859999999 podStartE2EDuration="5.409064671s" totalImagesPullingTime="3.683597985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:40.408964675 +0000 UTC m=+57.550390887" watchObservedRunningTime="2026-09-17 18:40:40.409064671 +0000 UTC m=+57.550490876"
Sep 17 18:40:40.429511 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.429377    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2z5j4" podStartSLOduration=2.89873794 podStartE2EDuration="6.429347099s" totalImagesPullingTime="3.530609159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:40.428062414 +0000 UTC m=+57.569488627" watchObservedRunningTime="2026-09-17 18:40:40.429347099 +0000 UTC m=+57.570773338"
Sep 17 18:40:40.453913 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:40.453845    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q88zg" podStartSLOduration=1.803330101 podStartE2EDuration="5.453826899s" totalImagesPullingTime="3.650496798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:40.452845992 +0000 UTC m=+57.594272204" watchObservedRunningTime="2026-09-17 18:40:40.453826899 +0000 UTC m=+57.595253111"
Sep 17 18:40:44.304500 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:44.304457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-22lq4"
Sep 17 18:40:50.400986 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.400956    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q88zg"
Sep 17 18:40:50.622214 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.622181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"]
Sep 17 18:40:50.625264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.625248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.642963 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.642938    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:40:50.642963 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.642961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:40:50.645807 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.645786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 18:40:50.645961 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.645858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 18:40:50.646235 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.646219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mnrdg\""
Sep 17 18:40:50.649035 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.649016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:50.672126 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.672067    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"]
Sep 17 18:40:50.730404 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.730369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:50.730404 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.730405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.730584 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.730425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.730584 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.730443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.730584 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.730556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxrm\" (UniqueName: \"kubernetes.io/projected/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-kube-api-access-4bxrm\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.733082 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.733064    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:40:50.743419 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.743390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b8b0027-07c0-4dde-b7af-6418d620ef89-original-pull-secret\") pod \"global-pull-secret-syncer-rzhx6\" (UID: \"1b8b0027-07c0-4dde-b7af-6418d620ef89\") " pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:50.804828 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.804776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzhx6"
Sep 17 18:40:50.831069 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.831227 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.831295 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.831295 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:50.831397 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:50.831397 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxrm\" (UniqueName: \"kubernetes.io/projected/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-kube-api-access-4bxrm\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.831826 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.831796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.833740 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.833713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.833850 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.833776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.834653 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.834632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:40:50.840077 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.840057    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:40:50.844307 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.844285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05ff87bb-d370-4f09-921b-a00049948b21-metrics-certs\") pod \"network-metrics-daemon-b9cf4\" (UID: \"05ff87bb-d370-4f09-921b-a00049948b21\") " pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:50.845954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.845933    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:40:50.846810 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.846787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxrm\" (UniqueName: \"kubernetes.io/projected/7d8e8d47-9129-47d3-8d46-9ca191d6a55c-kube-api-access-4bxrm\") pod \"prometheus-operator-7bd5d56d6-t8ms6\" (UID: \"7d8e8d47-9129-47d3-8d46-9ca191d6a55c\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:50.855024 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.854873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnd8\" (UniqueName: \"kubernetes.io/projected/39a299bf-7624-4e5d-a444-4d7580f04259-kube-api-access-6dnd8\") pod \"network-check-target-lpcg6\" (UID: \"39a299bf-7624-4e5d-a444-4d7580f04259\") " pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:50.934109 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:50.934030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"
Sep 17 18:40:51.085477 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.085449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzhx6"]
Sep 17 18:40:51.088145 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:51.088119    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8b0027_07c0_4dde_b7af_6418d620ef89.slice/crio-9e340f5f0eaa20b97dd985ccc235c204ebed8e381436add97fb667ff02119911 WatchSource:0}: Error finding container 9e340f5f0eaa20b97dd985ccc235c204ebed8e381436add97fb667ff02119911: Status 404 returned error can't find the container with id 9e340f5f0eaa20b97dd985ccc235c204ebed8e381436add97fb667ff02119911
Sep 17 18:40:51.094989 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.094970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:40:51.102378 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.102361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9cf4"
Sep 17 18:40:51.102454 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.102374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:40:51.110769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.110751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:51.207629 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.207570    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6"]
Sep 17 18:40:51.209535 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:51.209511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8e8d47_9129_47d3_8d46_9ca191d6a55c.slice/crio-fe9cafb4229e707a6998ef5082d6fb5ca95061647aeac6b20b3da6c2374f55e3 WatchSource:0}: Error finding container fe9cafb4229e707a6998ef5082d6fb5ca95061647aeac6b20b3da6c2374f55e3: Status 404 returned error can't find the container with id fe9cafb4229e707a6998ef5082d6fb5ca95061647aeac6b20b3da6c2374f55e3
Sep 17 18:40:51.391464 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.391432    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lpcg6"]
Sep 17 18:40:51.394812 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:51.394772    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a299bf_7624_4e5d_a444_4d7580f04259.slice/crio-b7b9f9afba5e172e1f6e654b312d08b88b27422dd7129c09f0d842af1c60a5f9 WatchSource:0}: Error finding container b7b9f9afba5e172e1f6e654b312d08b88b27422dd7129c09f0d842af1c60a5f9: Status 404 returned error can't find the container with id b7b9f9afba5e172e1f6e654b312d08b88b27422dd7129c09f0d842af1c60a5f9
Sep 17 18:40:51.396406 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.396378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b9cf4"]
Sep 17 18:40:51.399300 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:51.399278    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ff87bb_d370_4f09_921b_a00049948b21.slice/crio-d143bb85e69a26ba7bcf1a2536fd629fb5f89dc7cb9abd76dcda3f7b03eb28a3 WatchSource:0}: Error finding container d143bb85e69a26ba7bcf1a2536fd629fb5f89dc7cb9abd76dcda3f7b03eb28a3: Status 404 returned error can't find the container with id d143bb85e69a26ba7bcf1a2536fd629fb5f89dc7cb9abd76dcda3f7b03eb28a3
Sep 17 18:40:51.435857 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.435825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6" event={"ID":"7d8e8d47-9129-47d3-8d46-9ca191d6a55c","Type":"ContainerStarted","Data":"fe9cafb4229e707a6998ef5082d6fb5ca95061647aeac6b20b3da6c2374f55e3"}
Sep 17 18:40:51.437123 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.437098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpcg6" event={"ID":"39a299bf-7624-4e5d-a444-4d7580f04259","Type":"ContainerStarted","Data":"b7b9f9afba5e172e1f6e654b312d08b88b27422dd7129c09f0d842af1c60a5f9"}
Sep 17 18:40:51.438265 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.438246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9cf4" event={"ID":"05ff87bb-d370-4f09-921b-a00049948b21","Type":"ContainerStarted","Data":"d143bb85e69a26ba7bcf1a2536fd629fb5f89dc7cb9abd76dcda3f7b03eb28a3"}
Sep 17 18:40:51.439289 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:51.439265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzhx6" event={"ID":"1b8b0027-07c0-4dde-b7af-6418d620ef89","Type":"ContainerStarted","Data":"9e340f5f0eaa20b97dd985ccc235c204ebed8e381436add97fb667ff02119911"}
Sep 17 18:40:53.452545 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.452434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9cf4" event={"ID":"05ff87bb-d370-4f09-921b-a00049948b21","Type":"ContainerStarted","Data":"d80a7f7b4043bf216a8dd9acd134ef24180bca934a3ff9d78374fd9d2176f88c"}
Sep 17 18:40:53.453031 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.452581    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9cf4" event={"ID":"05ff87bb-d370-4f09-921b-a00049948b21","Type":"ContainerStarted","Data":"97c2bda77f2cceb382b30cf65694fb4182f36382fb917db8787796b69c377b97"}
Sep 17 18:40:53.455171 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.455081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6" event={"ID":"7d8e8d47-9129-47d3-8d46-9ca191d6a55c","Type":"ContainerStarted","Data":"d79e1901bb5440a6df092e7a6462692cb1b7cb6a558d0130c457656764d74f09"}
Sep 17 18:40:53.455171 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.455140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6" event={"ID":"7d8e8d47-9129-47d3-8d46-9ca191d6a55c","Type":"ContainerStarted","Data":"4af15c831feb42076eb295efd76bf40a796e991a567b27a82f61f523d5582908"}
Sep 17 18:40:53.522647 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.522583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-t8ms6" podStartSLOduration=1.812475182 podStartE2EDuration="3.5225584s" totalImagesPullingTime="1.710083218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:53.521177685 +0000 UTC m=+70.662603897" watchObservedRunningTime="2026-09-17 18:40:53.5225584 +0000 UTC m=+70.663984616"
Sep 17 18:40:53.523259 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:53.523223    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b9cf4" podStartSLOduration=67.003653601 podStartE2EDuration="1m8.523211056s" totalImagesPullingTime="1.519557455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:53.489486106 +0000 UTC m=+70.630912318" watchObservedRunningTime="2026-09-17 18:40:53.523211056 +0000 UTC m=+70.664637268"
Sep 17 18:40:55.132632 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.132496    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8d4xc"]
Sep 17 18:40:55.136086 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.136062    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.139177 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.139159    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:55.139409 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.139386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbjrk\""
Sep 17 18:40:55.139552 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.139529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:55.139654 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.139537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:55.165662 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165781 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165781 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-textfile\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165868 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-wtmp\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165868 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-root\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165982 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.165982 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-metrics-client-ca\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.166045 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.165979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-sys\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.166045 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.166003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gbb\" (UniqueName: \"kubernetes.io/projected/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-kube-api-access-82gbb\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267334 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-textfile\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-wtmp\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-root\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267431    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-root\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-metrics-client-ca\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-sys\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gbb\" (UniqueName: \"kubernetes.io/projected/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-kube-api-access-82gbb\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-wtmp\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-sys\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.267676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-textfile\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.267779 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:55.267741    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 18:40:55.268144 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:55.267799    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls podName:4a608575-01b3-40bd-b73a-ee9b1ba68c9e nodeName:}" failed. No retries permitted until 2026-09-17 18:40:55.767779553 +0000 UTC m=+72.909205742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls") pod "node-exporter-8d4xc" (UID: "4a608575-01b3-40bd-b73a-ee9b1ba68c9e") : secret "node-exporter-tls" not found
Sep 17 18:40:55.268219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.268184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.268288 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.268265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-metrics-client-ca\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.270296 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.270270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.278425 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.278399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gbb\" (UniqueName: \"kubernetes.io/projected/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-kube-api-access-82gbb\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.314851 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.314821    2561 patch_prober.go:28] interesting pod/image-registry-74bc5b974-xsm24 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:40:55.315062 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.314869    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74bc5b974-xsm24" podUID="5e1253ab-f0d2-407d-b53e-2ab940f3317a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:55.772427 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.772387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:55.775148 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:55.775123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a608575-01b3-40bd-b73a-ee9b1ba68c9e-node-exporter-tls\") pod \"node-exporter-8d4xc\" (UID: \"4a608575-01b3-40bd-b73a-ee9b1ba68c9e\") " pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:56.046482 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.046461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8d4xc"
Sep 17 18:40:56.058365 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:56.058333    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a608575_01b3_40bd_b73a_ee9b1ba68c9e.slice/crio-cee3a0b174098514dc829fe75cb12aa56803529426a177fc378b854435803e3e WatchSource:0}: Error finding container cee3a0b174098514dc829fe75cb12aa56803529426a177fc378b854435803e3e: Status 404 returned error can't find the container with id cee3a0b174098514dc829fe75cb12aa56803529426a177fc378b854435803e3e
Sep 17 18:40:56.183865 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.183829    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:40:56.186976 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.186946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.193244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:40:56.193244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:40:56.193244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:40:56.193244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193198    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:40:56.193516 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193366    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:40:56.193516 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.193442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:40:56.194784 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.194249    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:40:56.194784 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.194445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:40:56.194784 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.194628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-szqcq\""
Sep 17 18:40:56.194784 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.194649    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:40:56.219610 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.219546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:40:56.275622 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrdq\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-kube-api-access-vfrdq\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-config-out\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.275785 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.276120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.275954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-web-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.376760 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.376711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.376949 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.376770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.376949 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.376802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.376949 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.376830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.376949 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.376853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-web-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377290 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrdq\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-kube-api-access-vfrdq\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-config-out\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377846 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377846 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.377846 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.377482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.378579 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.378267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.379194 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:56.378923    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 18:40:56.379194 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:40:56.378992    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls podName:ad880bff-d546-4361-a1f0-594ec9783d3e nodeName:}" failed. No retries permitted until 2026-09-17 18:40:56.878974184 +0000 UTC m=+74.020400388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ad880bff-d546-4361-a1f0-594ec9783d3e") : secret "alertmanager-main-tls" not found
Sep 17 18:40:56.379194 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.379054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad880bff-d546-4361-a1f0-594ec9783d3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.380495 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.380467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-web-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.380721 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.380660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.380911 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.380866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.381192 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.381175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.381957 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.381935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad880bff-d546-4361-a1f0-594ec9783d3e-config-out\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.382114 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.382094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.382242 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.382224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.382431 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.382411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.389505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.389478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrdq\" (UniqueName: \"kubernetes.io/projected/ad880bff-d546-4361-a1f0-594ec9783d3e-kube-api-access-vfrdq\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.469227 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.469194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lpcg6" event={"ID":"39a299bf-7624-4e5d-a444-4d7580f04259","Type":"ContainerStarted","Data":"bb1e6775b02b4d09e1f68ef23834d70a5bc00b040dec88e5c1a7745a6e89dcfd"}
Sep 17 18:40:56.469400 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.469307    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:40:56.470849 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.470825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzhx6" event={"ID":"1b8b0027-07c0-4dde-b7af-6418d620ef89","Type":"ContainerStarted","Data":"443a4fb0b882a52b9d1607a53d421d79bc72311385dcb7a49b4ce0dcb03aea68"}
Sep 17 18:40:56.472107 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.472086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d4xc" event={"ID":"4a608575-01b3-40bd-b73a-ee9b1ba68c9e","Type":"ContainerStarted","Data":"cee3a0b174098514dc829fe75cb12aa56803529426a177fc378b854435803e3e"}
Sep 17 18:40:56.485633 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.485576    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lpcg6" podStartSLOduration=66.842249623 podStartE2EDuration="1m11.485557531s" totalImagesPullingTime="4.643307908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:56.484818627 +0000 UTC m=+73.626244842" watchObservedRunningTime="2026-09-17 18:40:56.485557531 +0000 UTC m=+73.626983744"
Sep 17 18:40:56.881718 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.881683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:56.884426 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:56.884398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad880bff-d546-4361-a1f0-594ec9783d3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad880bff-d546-4361-a1f0-594ec9783d3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:57.097055 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.097019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:40:57.379308 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.379256    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rzhx6" podStartSLOduration=67.42526292 podStartE2EDuration="1m12.379238377s" totalImagesPullingTime="4.953975457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:56.507555489 +0000 UTC m=+73.648981702" watchObservedRunningTime="2026-09-17 18:40:57.379238377 +0000 UTC m=+74.520664588"
Sep 17 18:40:57.379758 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.379674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:40:57.382532 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:40:57.382504    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad880bff_d546_4361_a1f0_594ec9783d3e.slice/crio-cc3554fd6caa0d5b750346a6815ca724b46013a1fca7a982c3439455ea7add16 WatchSource:0}: Error finding container cc3554fd6caa0d5b750346a6815ca724b46013a1fca7a982c3439455ea7add16: Status 404 returned error can't find the container with id cc3554fd6caa0d5b750346a6815ca724b46013a1fca7a982c3439455ea7add16
Sep 17 18:40:57.385227 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.385204    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74bc5b974-xsm24"
Sep 17 18:40:57.477684 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.477596    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a608575-01b3-40bd-b73a-ee9b1ba68c9e" containerID="798f32928cc8f8c6ab51386178271ab30447e6380315f5c6b0e9b5a01bedeac0" exitCode=0
Sep 17 18:40:57.477827 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.477681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d4xc" event={"ID":"4a608575-01b3-40bd-b73a-ee9b1ba68c9e","Type":"ContainerDied","Data":"798f32928cc8f8c6ab51386178271ab30447e6380315f5c6b0e9b5a01bedeac0"}
Sep 17 18:40:57.479036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:57.479010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"cc3554fd6caa0d5b750346a6815ca724b46013a1fca7a982c3439455ea7add16"}
Sep 17 18:40:58.484233 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:58.484202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d4xc" event={"ID":"4a608575-01b3-40bd-b73a-ee9b1ba68c9e","Type":"ContainerStarted","Data":"d87326eb7462342b8c0fc7d576284ff41f2b716410624c93c91b6a5022d91711"}
Sep 17 18:40:58.484619 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:58.484250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d4xc" event={"ID":"4a608575-01b3-40bd-b73a-ee9b1ba68c9e","Type":"ContainerStarted","Data":"bc75447ae2e67145d545b1f83f5d2c6768c54cc1e86d5d6bca0543e64cc00ac9"}
Sep 17 18:40:58.485863 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:58.485840    2561 generic.go:356] "Generic (PLEG): container finished" podID="ad880bff-d546-4361-a1f0-594ec9783d3e" containerID="d3cce705c706d5d12e0362374a1fd049f39529ce3b85cf7ea1ac6e3fed04b15d" exitCode=0
Sep 17 18:40:58.485959 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:58.485909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerDied","Data":"d3cce705c706d5d12e0362374a1fd049f39529ce3b85cf7ea1ac6e3fed04b15d"}
Sep 17 18:40:58.514116 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:58.514064    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8d4xc" podStartSLOduration=2.587366433 podStartE2EDuration="3.514046992s" totalImagesPullingTime="856.607316ms" totalInitContainerRuntime="70.073243ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:58.513179889 +0000 UTC m=+75.654606098" watchObservedRunningTime="2026-09-17 18:40:58.514046992 +0000 UTC m=+75.655473203"
Sep 17 18:40:59.455679 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.455646    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6589774446-4gl4j"]
Sep 17 18:40:59.457556 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.457541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.460424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.460401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:40:59.461328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.461310    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7qhnqgc5ukjgn\""
Sep 17 18:40:59.461615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.461603    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:40:59.462100 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.462075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:40:59.462226 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.462202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5bn4t\""
Sep 17 18:40:59.462429 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.462413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:40:59.472721 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.472699    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6589774446-4gl4j"]
Sep 17 18:40:59.502678 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.502678 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-metrics-server-audit-profiles\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.503058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mxl\" (UniqueName: \"kubernetes.io/projected/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-kube-api-access-b5mxl\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.503058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-client-certs\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.503058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-client-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.503058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-tls\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.503058 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.502873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-audit-log\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604106 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-client-certs\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604106 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-client-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-tls\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-audit-log\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.604324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-metrics-server-audit-profiles\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.605193 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mxl\" (UniqueName: \"kubernetes.io/projected/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-kube-api-access-b5mxl\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.605193 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.604775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-audit-log\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.605472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.605447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.605972 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.605949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-metrics-server-audit-profiles\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.607045 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.607019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-client-ca-bundle\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.607200 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.607181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-tls\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.607446 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.607421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-secret-metrics-server-client-certs\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.615637 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.615609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mxl\" (UniqueName: \"kubernetes.io/projected/42c40e78-8f7a-4dc3-b22e-05d13f41d58c-kube-api-access-b5mxl\") pod \"metrics-server-6589774446-4gl4j\" (UID: \"42c40e78-8f7a-4dc3-b22e-05d13f41d58c\") " pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.766641 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.766545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:40:59.864474 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.864430    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"]
Sep 17 18:40:59.866647 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.866622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:40:59.870211 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.870188    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tzn7z\""
Sep 17 18:40:59.870521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.870470    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 18:40:59.893317 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.893289    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"]
Sep 17 18:40:59.906833 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.906811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cpq\" (UniqueName: \"kubernetes.io/projected/6e78f413-dd78-415b-bce5-d90bdbaeae37-kube-api-access-25cpq\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:40:59.906959 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:40:59.906843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.008098 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.008073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cpq\" (UniqueName: \"kubernetes.io/projected/6e78f413-dd78-415b-bce5-d90bdbaeae37-kube-api-access-25cpq\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.008174 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.008126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.008308 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:41:00.008290    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 18:41:00.008375 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:41:00.008364    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert podName:6e78f413-dd78-415b-bce5-d90bdbaeae37 nodeName:}" failed. No retries permitted until 2026-09-17 18:41:00.508341732 +0000 UTC m=+77.649767931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-jm7w9" (UID: "6e78f413-dd78-415b-bce5-d90bdbaeae37") : secret "monitoring-plugin-cert" not found
Sep 17 18:41:00.016514 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.016481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cpq\" (UniqueName: \"kubernetes.io/projected/6e78f413-dd78-415b-bce5-d90bdbaeae37-kube-api-access-25cpq\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.197229 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:41:00.197186    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c40e78_8f7a_4dc3_b22e_05d13f41d58c.slice/crio-0e352f4c649c6b5c92ff7378a6b211f90eeb8c224cfa4be19317cff584d390fb WatchSource:0}: Error finding container 0e352f4c649c6b5c92ff7378a6b211f90eeb8c224cfa4be19317cff584d390fb: Status 404 returned error can't find the container with id 0e352f4c649c6b5c92ff7378a6b211f90eeb8c224cfa4be19317cff584d390fb
Sep 17 18:41:00.199310 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.198923    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6589774446-4gl4j"]
Sep 17 18:41:00.496512 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.496467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"25295db3fdc0265b5bbeb06566850d59979727af03b3d11db59e9c6e8e7b4185"}
Sep 17 18:41:00.496666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.496556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"95b1232c74b3baf2120e3e1ea28b6b1df668f5c4b203bae9dab1f973ed4e7a54"}
Sep 17 18:41:00.496666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.496593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"e3774ef502fbb3d570d2f367c0693bbff090114ad9e803e3e06a3b6a0bc70281"}
Sep 17 18:41:00.496666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.496619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"0b2fa91206e102ee54984ba9c79075ee0947ed77fb665a90be241ef143b49041"}
Sep 17 18:41:00.496666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.496649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"62f8ca1c7335c1d0b81c6194fd5adce757e205d2f81d414c80a386c1a3ffc121"}
Sep 17 18:41:00.498154 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.498124    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6589774446-4gl4j" event={"ID":"42c40e78-8f7a-4dc3-b22e-05d13f41d58c","Type":"ContainerStarted","Data":"0e352f4c649c6b5c92ff7378a6b211f90eeb8c224cfa4be19317cff584d390fb"}
Sep 17 18:41:00.513645 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.513615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.516125 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.516108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e78f413-dd78-415b-bce5-d90bdbaeae37-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-jm7w9\" (UID: \"6e78f413-dd78-415b-bce5-d90bdbaeae37\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:00.778205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:00.778120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:01.062958 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.062922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"]
Sep 17 18:41:01.066229 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:41:01.066200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e78f413_dd78_415b_bce5_d90bdbaeae37.slice/crio-02acda67b3f3683faa7eb4a1b70ad60030d7479d0ab90092b522c15e65eb34bd WatchSource:0}: Error finding container 02acda67b3f3683faa7eb4a1b70ad60030d7479d0ab90092b522c15e65eb34bd: Status 404 returned error can't find the container with id 02acda67b3f3683faa7eb4a1b70ad60030d7479d0ab90092b522c15e65eb34bd
Sep 17 18:41:01.515832 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.515790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad880bff-d546-4361-a1f0-594ec9783d3e","Type":"ContainerStarted","Data":"df016d4ce81c51609999757735b0c911019007c57699b9ad916267a3f2a74825"}
Sep 17 18:41:01.517618 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.517577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9" event={"ID":"6e78f413-dd78-415b-bce5-d90bdbaeae37","Type":"ContainerStarted","Data":"02acda67b3f3683faa7eb4a1b70ad60030d7479d0ab90092b522c15e65eb34bd"}
Sep 17 18:41:01.563650 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.563608    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:01.569715 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.569672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.576151 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.576123    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 18:41:01.578991 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.577925    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.428781049 podStartE2EDuration="5.577887848s" totalImagesPullingTime="3.133394406s" totalInitContainerRuntime="15.712393ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:01.569962885 +0000 UTC m=+78.711389096" watchObservedRunningTime="2026-09-17 18:41:01.577887848 +0000 UTC m=+78.719314060"
Sep 17 18:41:01.580423 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 18:41:01.580541 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 18:41:01.580541 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580513    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-83f4ho0aogvk1\""
Sep 17 18:41:01.580671 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580557    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 18:41:01.580833 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580813    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 18:41:01.580940 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580918    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 18:41:01.581036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.580970    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 18:41:01.581096 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 18:41:01.581164 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 18:41:01.581164 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 18:41:01.581264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 18:41:01.581264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kcbnb\""
Sep 17 18:41:01.581359 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.581298    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 18:41:01.583564 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.583534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 18:41:01.592922 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.590447    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 18:41:01.601337 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.601127    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:01.622434 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622615 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622866 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.622994 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623046 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.622987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623100 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623100 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623204 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623204 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623204 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623345 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623345 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623878 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.623878 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.624067 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.623960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tmx\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-kube-api-access-h4tmx\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725409 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725409 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725661 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725661 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.725769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.725976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726021 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tmx\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-kube-api-access-h4tmx\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.726838 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.726645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.727498 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.727191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.727498 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.727493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.727685 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.727568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.728260 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.728231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.728641 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.728617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.728828 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.728804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e13aa3db-d679-465d-8a56-ec68b30f70bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.729193 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.729169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.730366 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.730136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.730366 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.730316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e13aa3db-d679-465d-8a56-ec68b30f70bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.730711 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.730545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.731011 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.730975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.731631 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.731588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.732089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.732068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.732156 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.732104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.732204 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.732149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.732763 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.732746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.732933 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.732828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-config\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.733435 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.733410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e13aa3db-d679-465d-8a56-ec68b30f70bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.735126 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.735103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tmx\" (UniqueName: \"kubernetes.io/projected/e13aa3db-d679-465d-8a56-ec68b30f70bc-kube-api-access-h4tmx\") pod \"prometheus-k8s-0\" (UID: \"e13aa3db-d679-465d-8a56-ec68b30f70bc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:01.883457 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:01.883424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:02.182571 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.182542    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 18:41:02.183511 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:41:02.183481    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13aa3db_d679_465d_8a56_ec68b30f70bc.slice/crio-4739a5f337f75c19e8a20a6ad12b35c78d4cf0c08016840544098ae10c007120 WatchSource:0}: Error finding container 4739a5f337f75c19e8a20a6ad12b35c78d4cf0c08016840544098ae10c007120: Status 404 returned error can't find the container with id 4739a5f337f75c19e8a20a6ad12b35c78d4cf0c08016840544098ae10c007120
Sep 17 18:41:02.524444 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.524354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6589774446-4gl4j" event={"ID":"42c40e78-8f7a-4dc3-b22e-05d13f41d58c","Type":"ContainerStarted","Data":"4b1241a1878b1c1f379c13ef1c4c0fbc980739f787a8f0cc43dc5be68807eb48"}
Sep 17 18:41:02.526287 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.526261    2561 generic.go:356] "Generic (PLEG): container finished" podID="e13aa3db-d679-465d-8a56-ec68b30f70bc" containerID="626384013a20d9ca764975f7dee7ea66a1ac7b65e813bea95a3a01bbd9cfd9cd" exitCode=0
Sep 17 18:41:02.526406 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.526346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerDied","Data":"626384013a20d9ca764975f7dee7ea66a1ac7b65e813bea95a3a01bbd9cfd9cd"}
Sep 17 18:41:02.526453 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.526425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"4739a5f337f75c19e8a20a6ad12b35c78d4cf0c08016840544098ae10c007120"}
Sep 17 18:41:02.553667 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:02.553617    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6589774446-4gl4j" podStartSLOduration=1.991892221 podStartE2EDuration="3.553601443s" totalImagesPullingTime="1.561709222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:02.552010855 +0000 UTC m=+79.693437059" watchObservedRunningTime="2026-09-17 18:41:02.553601443 +0000 UTC m=+79.695027655"
Sep 17 18:41:03.532547 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:03.532506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9" event={"ID":"6e78f413-dd78-415b-bce5-d90bdbaeae37","Type":"ContainerStarted","Data":"33490b3786073f4bd23d6e566c7e903d628431bb00e8cdaa221547c7bb6c7ec0"}
Sep 17 18:41:03.553481 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:03.553425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9" podStartSLOduration=2.316305845 podStartE2EDuration="4.553403783s" totalImagesPullingTime="2.237097938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:03.552226498 +0000 UTC m=+80.693652709" watchObservedRunningTime="2026-09-17 18:41:03.553403783 +0000 UTC m=+80.694829996"
Sep 17 18:41:04.536832 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:04.536799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:04.542426 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:04.542405    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-jm7w9"
Sep 17 18:41:06.547596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:06.547553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"7fcf084103376b993e9137f3924a5a81eee2439075ca8f3c861a750e5ffaeaa0"}
Sep 17 18:41:06.548033 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:06.547644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"f2400b5d34bf1dce3c7ddd38ce6648f2e0391efc4e7c4d66843c33a3ec6f3717"}
Sep 17 18:41:08.561544 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:08.561508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"b728af1cc156f717a89dfd3a71e3091209d46b91cebefa9aa2ac6864eddf2060"}
Sep 17 18:41:08.561941 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:08.561608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"3e88a08b672039dd73bf692ba6662f790df1e7eae9a08b0b5b74b9d87f83ddd6"}
Sep 17 18:41:08.561941 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:08.561637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"0aab8aabc9cd97faa37c96d14badfacaabb9a003ddde159ddd775f100d9696b9"}
Sep 17 18:41:08.561941 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:08.561677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e13aa3db-d679-465d-8a56-ec68b30f70bc","Type":"ContainerStarted","Data":"f82068a3fd16bbb901c4bbca5de7be22bf27b4902393a63a13705e03a30af145"}
Sep 17 18:41:08.629884 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:08.629827    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.58883671 podStartE2EDuration="7.629810963s" totalImagesPullingTime="5.022134051s" totalInitContainerRuntime="18.840202ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:08.625756555 +0000 UTC m=+85.767182766" watchObservedRunningTime="2026-09-17 18:41:08.629810963 +0000 UTC m=+85.771237175"
Sep 17 18:41:11.884141 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:11.884095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:41:19.767655 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:19.767618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:41:19.768142 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:19.767691    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:41:27.481522 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:27.481493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lpcg6"
Sep 17 18:41:39.772353 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:39.772324    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:41:39.776013 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:41:39.775990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6589774446-4gl4j"
Sep 17 18:42:01.884690 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:42:01.884645    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:42:01.905688 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:42:01.905657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:42:02.785592 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:42:02.785552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 18:44:44.937906 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:44:44.937864    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:44:44.941538 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:44:44.941518    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:44:44.971754 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:44:44.971729    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:46:09.696692 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.696656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-phdsp"]
Sep 17 18:46:09.699713 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.699694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.703528 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.703495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 18:46:09.704264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.704247    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 18:46:09.704363 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.704253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qb4fr\""
Sep 17 18:46:09.722611 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.722589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-phdsp"]
Sep 17 18:46:09.867963 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.867883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76c5t\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-kube-api-access-76c5t\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.868134 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.867975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-bound-sa-token\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.963108 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.963026    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"]
Sep 17 18:46:09.966121 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.966093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:09.968413 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.968391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-bound-sa-token\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.968503 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.968453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-776k5\""
Sep 17 18:46:09.968503 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.968488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76c5t\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-kube-api-access-76c5t\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.973752 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.973721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"]
Sep 17 18:46:09.979293 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.979271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-bound-sa-token\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:09.979626 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:09.979607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76c5t\" (UniqueName: \"kubernetes.io/projected/7af3cb4c-dc7e-4fe8-a626-cca78374ac15-kube-api-access-76c5t\") pod \"cert-manager-55b4d954d5-phdsp\" (UID: \"7af3cb4c-dc7e-4fe8-a626-cca78374ac15\") " pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:10.030588 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.030558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-phdsp"
Sep 17 18:46:10.068876 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.068844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf5d\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-kube-api-access-6jf5d\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.069062 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.068921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.169966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.169927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf5d\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-kube-api-access-6jf5d\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.170121 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.170005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.178304 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.178269    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.178472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.178450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf5d\" (UniqueName: \"kubernetes.io/projected/5c00b057-d0d7-4fef-a309-c4410ddbf608-kube-api-access-6jf5d\") pod \"cert-manager-cainjector-559cdd8c87-89qlw\" (UID: \"5c00b057-d0d7-4fef-a309-c4410ddbf608\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.275263 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.275202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"
Sep 17 18:46:10.300176 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.300157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-phdsp"]
Sep 17 18:46:10.302530 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:46:10.302503    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af3cb4c_dc7e_4fe8_a626_cca78374ac15.slice/crio-f821585bbbcfead7f14e47456c056e63ea3a2ed78b4460248ef95e5336547381 WatchSource:0}: Error finding container f821585bbbcfead7f14e47456c056e63ea3a2ed78b4460248ef95e5336547381: Status 404 returned error can't find the container with id f821585bbbcfead7f14e47456c056e63ea3a2ed78b4460248ef95e5336547381
Sep 17 18:46:10.304551 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.304527    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:46:10.551683 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.551613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-89qlw"]
Sep 17 18:46:10.554136 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:46:10.554102    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c00b057_d0d7_4fef_a309_c4410ddbf608.slice/crio-2955181e3d2c0daeedb8b6bad621d945835c6c9c6341355130f5f6c2c136d8c4 WatchSource:0}: Error finding container 2955181e3d2c0daeedb8b6bad621d945835c6c9c6341355130f5f6c2c136d8c4: Status 404 returned error can't find the container with id 2955181e3d2c0daeedb8b6bad621d945835c6c9c6341355130f5f6c2c136d8c4
Sep 17 18:46:10.695359 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.695323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-phdsp" event={"ID":"7af3cb4c-dc7e-4fe8-a626-cca78374ac15","Type":"ContainerStarted","Data":"f821585bbbcfead7f14e47456c056e63ea3a2ed78b4460248ef95e5336547381"}
Sep 17 18:46:10.696561 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:10.696541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw" event={"ID":"5c00b057-d0d7-4fef-a309-c4410ddbf608","Type":"ContainerStarted","Data":"2955181e3d2c0daeedb8b6bad621d945835c6c9c6341355130f5f6c2c136d8c4"}
Sep 17 18:46:13.713256 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:13.713217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-phdsp" event={"ID":"7af3cb4c-dc7e-4fe8-a626-cca78374ac15","Type":"ContainerStarted","Data":"30f54bfa0c2325958740ecc4096812c8fd01f6ee7efbc400711e80b8df93c48d"}
Sep 17 18:46:13.715222 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:13.715194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw" event={"ID":"5c00b057-d0d7-4fef-a309-c4410ddbf608","Type":"ContainerStarted","Data":"ba545009bf8ce13ee1dfa46c806871b7c7c13d487865ce29cad272bfe82254ed"}
Sep 17 18:46:13.730889 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:13.730841    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-phdsp" podStartSLOduration=1.547191497 podStartE2EDuration="4.730825237s" totalImagesPullingTime="3.18363374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:13.729181653 +0000 UTC m=+390.870607868" watchObservedRunningTime="2026-09-17 18:46:13.730825237 +0000 UTC m=+390.872251449"
Sep 17 18:46:13.749101 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:13.748999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-89qlw" podStartSLOduration=1.8052702919999999 podStartE2EDuration="4.748982105s" totalImagesPullingTime="2.943711813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:13.748045687 +0000 UTC m=+390.889471899" watchObservedRunningTime="2026-09-17 18:46:13.748982105 +0000 UTC m=+390.890408298"
Sep 17 18:46:35.721363 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.721332    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"]
Sep 17 18:46:35.725838 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.725814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.729417 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.729381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:46:35.729591 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.729565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-5jwsc\""
Sep 17 18:46:35.729705 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.729477    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:46:35.730433 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.730414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:46:35.737115 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.737094    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"]
Sep 17 18:46:35.770415 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a413b68-67ab-4c5f-a450-cd459e093103-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770523 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770523 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770523 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770642 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770642 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770642 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a413b68-67ab-4c5f-a450-cd459e093103-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770752 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt669\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-kube-api-access-xt669\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.770752 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.770677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.871827 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.871797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a413b68-67ab-4c5f-a450-cd459e093103-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.871827 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.871828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt669\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-kube-api-access-xt669\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872041 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.871848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872041 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.871963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a413b68-67ab-4c5f-a450-cd459e093103-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872124 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872124 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872217 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872217 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872312 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872482 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872627 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872627 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872627 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a413b68-67ab-4c5f-a450-cd459e093103-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.872805 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.872643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.874517 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.874496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a413b68-67ab-4c5f-a450-cd459e093103-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.874727 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.874711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a413b68-67ab-4c5f-a450-cd459e093103-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.880709 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.880688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:35.881162 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:35.881128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt669\" (UniqueName: \"kubernetes.io/projected/5a413b68-67ab-4c5f-a450-cd459e093103-kube-api-access-xt669\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k\" (UID: \"5a413b68-67ab-4c5f-a450-cd459e093103\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:36.037187 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:36.037107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:36.312147 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:36.311972    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"]
Sep 17 18:46:36.314861 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:46:36.314836    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a413b68_67ab_4c5f_a450_cd459e093103.slice/crio-e4b3058b67cab8f8735e62352c3e0a6541240168d2392be298c2b8437f04ad7f WatchSource:0}: Error finding container e4b3058b67cab8f8735e62352c3e0a6541240168d2392be298c2b8437f04ad7f: Status 404 returned error can't find the container with id e4b3058b67cab8f8735e62352c3e0a6541240168d2392be298c2b8437f04ad7f
Sep 17 18:46:36.804337 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:36.804305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" event={"ID":"5a413b68-67ab-4c5f-a450-cd459e093103","Type":"ContainerStarted","Data":"e4b3058b67cab8f8735e62352c3e0a6541240168d2392be298c2b8437f04ad7f"}
Sep 17 18:46:39.005723 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:39.005683    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:46:39.006135 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:39.005780    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:46:39.006135 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:39.005828    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:46:39.818490 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:39.818404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" event={"ID":"5a413b68-67ab-4c5f-a450-cd459e093103","Type":"ContainerStarted","Data":"21d7e826dc312fb934adc28289990ce28ec3752500e3ab92370cd42ba0aa6c23"}
Sep 17 18:46:39.841332 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:39.841283    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podStartSLOduration=2.153195395 podStartE2EDuration="4.841266964s" totalImagesPullingTime="2.688071569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:39.839658847 +0000 UTC m=+416.981085095" watchObservedRunningTime="2026-09-17 18:46:39.841266964 +0000 UTC m=+416.982693175"
Sep 17 18:46:40.038289 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:40.038256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:40.042814 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:40.042788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:40.822512 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:40.822476    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:40.823630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:40.823606    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:46:46.505755 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.505717    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-875gt"]
Sep 17 18:46:46.509175 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.509149    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.512139 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.512113    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fn5xz\""
Sep 17 18:46:46.512269 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.512151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:46:46.513038 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.513003    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 18:46:46.513145 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.513014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:46:46.513220 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.513164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 18:46:46.513364 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.513324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 18:46:46.534163 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.534132    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-875gt"]
Sep 17 18:46:46.670313 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.670281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-metrics-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.670504 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.670321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14a7b116-59d4-4e29-978c-a2a450a401f0-manager-config\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.670504 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.670345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.670504 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.670371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7twq\" (UniqueName: \"kubernetes.io/projected/14a7b116-59d4-4e29-978c-a2a450a401f0-kube-api-access-c7twq\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.771782 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.771689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-metrics-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.771782 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.771730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14a7b116-59d4-4e29-978c-a2a450a401f0-manager-config\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.771782 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.771771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.772079 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.771802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7twq\" (UniqueName: \"kubernetes.io/projected/14a7b116-59d4-4e29-978c-a2a450a401f0-kube-api-access-c7twq\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.772437 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.772418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14a7b116-59d4-4e29-978c-a2a450a401f0-manager-config\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.774433 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.774411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.774541 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.774438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14a7b116-59d4-4e29-978c-a2a450a401f0-metrics-cert\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.781200 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.781177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7twq\" (UniqueName: \"kubernetes.io/projected/14a7b116-59d4-4e29-978c-a2a450a401f0-kube-api-access-c7twq\") pod \"lws-controller-manager-54799454dd-875gt\" (UID: \"14a7b116-59d4-4e29-978c-a2a450a401f0\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:46.818248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:46.818215    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:46:47.110834 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:47.110798    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-875gt"]
Sep 17 18:46:47.115085 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:46:47.115055    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a7b116_59d4_4e29_978c_a2a450a401f0.slice/crio-9bc2b0bb3ed5326ccf687c6bedf84d4c5e5d091740c3348d0061306ac917a70f WatchSource:0}: Error finding container 9bc2b0bb3ed5326ccf687c6bedf84d4c5e5d091740c3348d0061306ac917a70f: Status 404 returned error can't find the container with id 9bc2b0bb3ed5326ccf687c6bedf84d4c5e5d091740c3348d0061306ac917a70f
Sep 17 18:46:47.849328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:47.849295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt" event={"ID":"14a7b116-59d4-4e29-978c-a2a450a401f0","Type":"ContainerStarted","Data":"9bc2b0bb3ed5326ccf687c6bedf84d4c5e5d091740c3348d0061306ac917a70f"}
Sep 17 18:46:49.859165 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:49.859124    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt" event={"ID":"14a7b116-59d4-4e29-978c-a2a450a401f0","Type":"ContainerStarted","Data":"8765dec46ba2841099b4fdf4ac1f7a469a007cba4396181af36829556e5993e4"}
Sep 17 18:46:49.859526 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:46:49.859262    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:47:00.866042 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:00.866010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt"
Sep 17 18:47:00.883603 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:00.883551    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54799454dd-875gt" podStartSLOduration=12.309980512 podStartE2EDuration="14.883534152s" totalImagesPullingTime="2.57355364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:49.887570706 +0000 UTC m=+427.028996920" watchObservedRunningTime="2026-09-17 18:47:00.883534152 +0000 UTC m=+438.024960363"
Sep 17 18:47:19.466865 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.466832    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"]
Sep 17 18:47:19.475947 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.475921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.479205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.479183    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:47:19.479205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.479192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g9456\""
Sep 17 18:47:19.479639 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.479619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:47:19.489470 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.489448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"]
Sep 17 18:47:19.559360 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.559313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.559360 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.559370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2fc\" (UniqueName: \"kubernetes.io/projected/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-kube-api-access-gr2fc\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.660170 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.660124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.660170 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.660182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2fc\" (UniqueName: \"kubernetes.io/projected/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-kube-api-access-gr2fc\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.660517 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.660498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.672915 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.672867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2fc\" (UniqueName: \"kubernetes.io/projected/24e2a03d-efdf-4aed-ba4f-b634ff5d050f-kube-api-access-gr2fc\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dlnhg\" (UID: \"24e2a03d-efdf-4aed-ba4f-b634ff5d050f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:19.786301 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:19.786225    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:20.066975 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:20.066949    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"]
Sep 17 18:47:20.069084 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:47:20.069059    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e2a03d_efdf_4aed_ba4f_b634ff5d050f.slice/crio-bed6f7af5483fac4b48a18cb13dddd37d4d2cb2a1082440064aea7fae89ae710 WatchSource:0}: Error finding container bed6f7af5483fac4b48a18cb13dddd37d4d2cb2a1082440064aea7fae89ae710: Status 404 returned error can't find the container with id bed6f7af5483fac4b48a18cb13dddd37d4d2cb2a1082440064aea7fae89ae710
Sep 17 18:47:20.983650 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:20.983615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg" event={"ID":"24e2a03d-efdf-4aed-ba4f-b634ff5d050f","Type":"ContainerStarted","Data":"bed6f7af5483fac4b48a18cb13dddd37d4d2cb2a1082440064aea7fae89ae710"}
Sep 17 18:47:25.005007 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:25.004968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg" event={"ID":"24e2a03d-efdf-4aed-ba4f-b634ff5d050f","Type":"ContainerStarted","Data":"ed679f538212aa48513a0ed3b6079cbf1e07d1ea09aab866e0e1d82a973e051d"}
Sep 17 18:47:25.005396 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:25.005039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:47:25.033610 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:25.033553    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg" podStartSLOduration=1.870899452 podStartE2EDuration="6.033534764s" totalImagesPullingTime="4.162635312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:25.031023023 +0000 UTC m=+462.172449232" watchObservedRunningTime="2026-09-17 18:47:25.033534764 +0000 UTC m=+462.174960978"
Sep 17 18:47:36.012319 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:47:36.012274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dlnhg"
Sep 17 18:48:10.234062 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.234027    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:10.236795 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.236778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:10.240178 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.240156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jxhc4\""
Sep 17 18:48:10.250188 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.250164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:10.311223 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.311194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clj7\" (UniqueName: \"kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7\") pod \"authorino-7bcfd8f9cd-rsxcr\" (UID: \"77fe0aba-00b4-41ce-8ca3-c944d01534a6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:10.375476 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.375440    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:10.377934 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.377910    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:10.385509 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.385484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:10.411712 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.411682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clj7\" (UniqueName: \"kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7\") pod \"authorino-7bcfd8f9cd-rsxcr\" (UID: \"77fe0aba-00b4-41ce-8ca3-c944d01534a6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:10.420534 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.420510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clj7\" (UniqueName: \"kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7\") pod \"authorino-7bcfd8f9cd-rsxcr\" (UID: \"77fe0aba-00b4-41ce-8ca3-c944d01534a6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:10.512703 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.512619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmxt\" (UniqueName: \"kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt\") pod \"authorino-59c974bd48-8lzkq\" (UID: \"789a0cbe-1bdf-42db-adee-0ca729e43016\") " pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:10.546053 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.546022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:10.613591 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.613559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmxt\" (UniqueName: \"kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt\") pod \"authorino-59c974bd48-8lzkq\" (UID: \"789a0cbe-1bdf-42db-adee-0ca729e43016\") " pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:10.623507 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.623476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmxt\" (UniqueName: \"kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt\") pod \"authorino-59c974bd48-8lzkq\" (UID: \"789a0cbe-1bdf-42db-adee-0ca729e43016\") " pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:10.687512 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.687473    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:10.819230 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.819151    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:10.822495 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:48:10.822463    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77fe0aba_00b4_41ce_8ca3_c944d01534a6.slice/crio-9f362244a6ec7f9bfb6d9b98fd07fdfb2e9288f4ffc9dee47c5b46186ce786fd WatchSource:0}: Error finding container 9f362244a6ec7f9bfb6d9b98fd07fdfb2e9288f4ffc9dee47c5b46186ce786fd: Status 404 returned error can't find the container with id 9f362244a6ec7f9bfb6d9b98fd07fdfb2e9288f4ffc9dee47c5b46186ce786fd
Sep 17 18:48:10.957985 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:10.957948    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:10.961097 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:48:10.961068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod789a0cbe_1bdf_42db_adee_0ca729e43016.slice/crio-f0f577db52b869bd791d76cc52cbb8451b17681fcb3e9f9ad16d50ebed9426b1 WatchSource:0}: Error finding container f0f577db52b869bd791d76cc52cbb8451b17681fcb3e9f9ad16d50ebed9426b1: Status 404 returned error can't find the container with id f0f577db52b869bd791d76cc52cbb8451b17681fcb3e9f9ad16d50ebed9426b1
Sep 17 18:48:11.205877 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:11.205837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8lzkq" event={"ID":"789a0cbe-1bdf-42db-adee-0ca729e43016","Type":"ContainerStarted","Data":"f0f577db52b869bd791d76cc52cbb8451b17681fcb3e9f9ad16d50ebed9426b1"}
Sep 17 18:48:11.207112 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:11.207086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" event={"ID":"77fe0aba-00b4-41ce-8ca3-c944d01534a6","Type":"ContainerStarted","Data":"9f362244a6ec7f9bfb6d9b98fd07fdfb2e9288f4ffc9dee47c5b46186ce786fd"}
Sep 17 18:48:13.221442 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:13.221402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8lzkq" event={"ID":"789a0cbe-1bdf-42db-adee-0ca729e43016","Type":"ContainerStarted","Data":"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"}
Sep 17 18:48:13.223049 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:13.223020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" event={"ID":"77fe0aba-00b4-41ce-8ca3-c944d01534a6","Type":"ContainerStarted","Data":"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"}
Sep 17 18:48:13.238921 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:13.238860    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-8lzkq" podStartSLOduration=1.467256428 podStartE2EDuration="3.238839507s" totalImagesPullingTime="1.771583079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:13.238225649 +0000 UTC m=+510.379651868" watchObservedRunningTime="2026-09-17 18:48:13.238839507 +0000 UTC m=+510.380265717"
Sep 17 18:48:13.255198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:13.255148    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" podStartSLOduration=1.349663006 podStartE2EDuration="3.255129915s" totalImagesPullingTime="1.905466909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:13.25462514 +0000 UTC m=+510.396051352" watchObservedRunningTime="2026-09-17 18:48:13.255129915 +0000 UTC m=+510.396556126"
Sep 17 18:48:13.279428 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:13.279391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:15.231359 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:15.231310    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" podUID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" containerName="authorino" containerID="cri-o://8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c" gracePeriod=30
Sep 17 18:48:15.478165 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:15.478141    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:15.558853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:15.558760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2clj7\" (UniqueName: \"kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7\") pod \"77fe0aba-00b4-41ce-8ca3-c944d01534a6\" (UID: \"77fe0aba-00b4-41ce-8ca3-c944d01534a6\") "
Sep 17 18:48:15.561189 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:15.561160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7" (OuterVolumeSpecName: "kube-api-access-2clj7") pod "77fe0aba-00b4-41ce-8ca3-c944d01534a6" (UID: "77fe0aba-00b4-41ce-8ca3-c944d01534a6"). InnerVolumeSpecName "kube-api-access-2clj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:15.660172 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:15.660137    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clj7\" (UniqueName: \"kubernetes.io/projected/77fe0aba-00b4-41ce-8ca3-c944d01534a6-kube-api-access-2clj7\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:48:16.236398 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.236362    2561 generic.go:356] "Generic (PLEG): container finished" podID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" containerID="8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c" exitCode=0
Sep 17 18:48:16.236398 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.236398    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr"
Sep 17 18:48:16.236398 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.236402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" event={"ID":"77fe0aba-00b4-41ce-8ca3-c944d01534a6","Type":"ContainerDied","Data":"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"}
Sep 17 18:48:16.236975 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.236440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-rsxcr" event={"ID":"77fe0aba-00b4-41ce-8ca3-c944d01534a6","Type":"ContainerDied","Data":"9f362244a6ec7f9bfb6d9b98fd07fdfb2e9288f4ffc9dee47c5b46186ce786fd"}
Sep 17 18:48:16.236975 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.236458    2561 scope.go:117] "RemoveContainer" containerID="8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"
Sep 17 18:48:16.245452 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.245432    2561 scope.go:117] "RemoveContainer" containerID="8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"
Sep 17 18:48:16.245718 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:48:16.245688    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c\": container with ID starting with 8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c not found: ID does not exist" containerID="8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"
Sep 17 18:48:16.245775 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.245730    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c"} err="failed to get container status \"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c\": rpc error: code = NotFound desc = could not find container \"8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c\": container with ID starting with 8f30bcd3cfb9d66bea227e73604b763a04896700e14cdce0fea1a06dd440285c not found: ID does not exist"
Sep 17 18:48:16.257882 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.257855    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:16.260906 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:16.260871    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-rsxcr"]
Sep 17 18:48:17.087010 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:17.086980    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" path="/var/lib/kubelet/pods/77fe0aba-00b4-41ce-8ca3-c944d01534a6/volumes"
Sep 17 18:48:35.253572 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.253484    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gxlc4"]
Sep 17 18:48:35.254044 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.253807    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" containerName="authorino"
Sep 17 18:48:35.254044 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.253819    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" containerName="authorino"
Sep 17 18:48:35.254044 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.253888    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fe0aba-00b4-41ce-8ca3-c944d01534a6" containerName="authorino"
Sep 17 18:48:35.256559 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.256536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.259092 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.259065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 18:48:35.272319 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.272289    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gxlc4"]
Sep 17 18:48:35.324598 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.324563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4k4\" (UniqueName: \"kubernetes.io/projected/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-kube-api-access-tf4k4\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.324784 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.324692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-tls-cert\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.425217 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.425186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-tls-cert\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.425424 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.425241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4k4\" (UniqueName: \"kubernetes.io/projected/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-kube-api-access-tf4k4\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.427880 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.427853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-tls-cert\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.435104 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.435081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4k4\" (UniqueName: \"kubernetes.io/projected/3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74-kube-api-access-tf4k4\") pod \"authorino-6d9898bd99-gxlc4\" (UID: \"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74\") " pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.566870 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.566771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gxlc4"
Sep 17 18:48:35.860050 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:35.860017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gxlc4"]
Sep 17 18:48:35.863548 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:48:35.863517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ddfb2f2_2d9b_4dbf_8083_7b11729e6e74.slice/crio-36affb44f895f981d949749d760ca40ed0100934df25d5043db104de9e0d28e5 WatchSource:0}: Error finding container 36affb44f895f981d949749d760ca40ed0100934df25d5043db104de9e0d28e5: Status 404 returned error can't find the container with id 36affb44f895f981d949749d760ca40ed0100934df25d5043db104de9e0d28e5
Sep 17 18:48:36.323616 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:36.323539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gxlc4" event={"ID":"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74","Type":"ContainerStarted","Data":"36affb44f895f981d949749d760ca40ed0100934df25d5043db104de9e0d28e5"}
Sep 17 18:48:37.329545 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.329507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gxlc4" event={"ID":"3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74","Type":"ContainerStarted","Data":"cbb4ea1454ab9a1ce9bae6ea39fdf2dd495fa29e1a321eb67f6f8f40e1e538f3"}
Sep 17 18:48:37.351945 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.351870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-gxlc4" podStartSLOduration=1.890511871 podStartE2EDuration="2.351852454s" totalImagesPullingTime="461.340583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:37.351374757 +0000 UTC m=+534.492800968" watchObservedRunningTime="2026-09-17 18:48:37.351852454 +0000 UTC m=+534.493278664"
Sep 17 18:48:37.423393 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.423359    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:37.423579 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.423557    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-8lzkq" podUID="789a0cbe-1bdf-42db-adee-0ca729e43016" containerName="authorino" containerID="cri-o://2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd" gracePeriod=30
Sep 17 18:48:37.667596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.667569    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:37.742659 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.742621    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kmxt\" (UniqueName: \"kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt\") pod \"789a0cbe-1bdf-42db-adee-0ca729e43016\" (UID: \"789a0cbe-1bdf-42db-adee-0ca729e43016\") "
Sep 17 18:48:37.744963 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.744935    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt" (OuterVolumeSpecName: "kube-api-access-6kmxt") pod "789a0cbe-1bdf-42db-adee-0ca729e43016" (UID: "789a0cbe-1bdf-42db-adee-0ca729e43016"). InnerVolumeSpecName "kube-api-access-6kmxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:37.843382 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:37.843287    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kmxt\" (UniqueName: \"kubernetes.io/projected/789a0cbe-1bdf-42db-adee-0ca729e43016-kube-api-access-6kmxt\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:48:38.335217 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.335185    2561 generic.go:356] "Generic (PLEG): container finished" podID="789a0cbe-1bdf-42db-adee-0ca729e43016" containerID="2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd" exitCode=0
Sep 17 18:48:38.335630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.335228    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8lzkq"
Sep 17 18:48:38.335630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.335267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8lzkq" event={"ID":"789a0cbe-1bdf-42db-adee-0ca729e43016","Type":"ContainerDied","Data":"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"}
Sep 17 18:48:38.335630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.335311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8lzkq" event={"ID":"789a0cbe-1bdf-42db-adee-0ca729e43016","Type":"ContainerDied","Data":"f0f577db52b869bd791d76cc52cbb8451b17681fcb3e9f9ad16d50ebed9426b1"}
Sep 17 18:48:38.335630 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.335328    2561 scope.go:117] "RemoveContainer" containerID="2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"
Sep 17 18:48:38.344327 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.344309    2561 scope.go:117] "RemoveContainer" containerID="2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"
Sep 17 18:48:38.344617 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:48:38.344601    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd\": container with ID starting with 2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd not found: ID does not exist" containerID="2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"
Sep 17 18:48:38.344658 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.344627    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd"} err="failed to get container status \"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd\": rpc error: code = NotFound desc = could not find container \"2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd\": container with ID starting with 2b5ae4209eee07e2a1ea935fb86f605335f63a820ae9fec44e17ec88b98f27dd not found: ID does not exist"
Sep 17 18:48:38.361337 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.361306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:38.364375 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:38.364350    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8lzkq"]
Sep 17 18:48:39.087093 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:48:39.087060    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789a0cbe-1bdf-42db-adee-0ca729e43016" path="/var/lib/kubelet/pods/789a0cbe-1bdf-42db-adee-0ca729e43016/volumes"
Sep 17 18:49:12.879942 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.879881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"]
Sep 17 18:49:12.880383 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.880218    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789a0cbe-1bdf-42db-adee-0ca729e43016" containerName="authorino"
Sep 17 18:49:12.880383 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.880230    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="789a0cbe-1bdf-42db-adee-0ca729e43016" containerName="authorino"
Sep 17 18:49:12.880383 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.880296    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="789a0cbe-1bdf-42db-adee-0ca729e43016" containerName="authorino"
Sep 17 18:49:12.883379 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.883355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:12.887236 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.887203    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:49:12.887356 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.887301    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:49:12.887356 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.887299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6w82v\""
Sep 17 18:49:12.887461 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.887358    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:49:12.895925 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:12.895875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"]
Sep 17 18:49:13.032783 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.032748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2rl\" (UniqueName: \"kubernetes.io/projected/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-kube-api-access-lz2rl\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.032974 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.032804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-cert\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.134120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.134028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2rl\" (UniqueName: \"kubernetes.io/projected/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-kube-api-access-lz2rl\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.134120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.134074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-cert\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.136648 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.136618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-cert\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.145919 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.145884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2rl\" (UniqueName: \"kubernetes.io/projected/0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb-kube-api-access-lz2rl\") pod \"llmisvc-controller-manager-6b569b69c4-pd6vx\" (UID: \"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb\") " pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.193954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.193923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:13.517072 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:13.516999    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"]
Sep 17 18:49:13.520099 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:49:13.520071    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0b1c61f4_72fa_4fd0_90e4_e31a6c27cebb.slice/crio-494d0d40520b8db193c50ccfd04e6b9a5990f30a2eb18c84acde1cb1420bbd51 WatchSource:0}: Error finding container 494d0d40520b8db193c50ccfd04e6b9a5990f30a2eb18c84acde1cb1420bbd51: Status 404 returned error can't find the container with id 494d0d40520b8db193c50ccfd04e6b9a5990f30a2eb18c84acde1cb1420bbd51
Sep 17 18:49:14.507509 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:14.507466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx" event={"ID":"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb","Type":"ContainerStarted","Data":"494d0d40520b8db193c50ccfd04e6b9a5990f30a2eb18c84acde1cb1420bbd51"}
Sep 17 18:49:16.519157 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:16.519117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx" event={"ID":"0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb","Type":"ContainerStarted","Data":"b34e38ff97af396b50f1c2f3d792666cf909cf54ee853e9d565833d1faca4383"}
Sep 17 18:49:16.519560 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:16.519322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:49:16.537911 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:16.537855    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx" podStartSLOduration=2.147334436 podStartE2EDuration="4.537839941s" totalImagesPullingTime="2.390505505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:16.535963752 +0000 UTC m=+573.677390154" watchObservedRunningTime="2026-09-17 18:49:16.537839941 +0000 UTC m=+573.679266182"
Sep 17 18:49:44.964447 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:44.964420    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:49:44.996642 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:44.996617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:49:47.526504 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:49:47.526471    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b569b69c4-pd6vx"
Sep 17 18:52:28.378292 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.378258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:52:28.380651 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.380629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.384273 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.384250    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:52:28.384411 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.384305    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-t6wn5\""
Sep 17 18:52:28.384411 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.384389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5rcdt\""
Sep 17 18:52:28.384411 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.384401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:52:28.384539 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.384484    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:52:28.393650 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.393628    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:52:28.495307 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.495270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzhc\" (UniqueName: \"kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.495467 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.495370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.596622 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.596579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzhc\" (UniqueName: \"kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.596827 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.596668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.599333 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.599304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.604502 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.604478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzhc\" (UniqueName: \"kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.691520 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.691445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:28.978747 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.978690    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:52:28.982026 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:52:28.982000    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2e77c1_b92a_4f14_8b74_623649b97e79.slice/crio-2a33f717500a560ad3fc3f63d7a324c21a453c8e1af31745971bc45963ad22e8 WatchSource:0}: Error finding container 2a33f717500a560ad3fc3f63d7a324c21a453c8e1af31745971bc45963ad22e8: Status 404 returned error can't find the container with id 2a33f717500a560ad3fc3f63d7a324c21a453c8e1af31745971bc45963ad22e8
Sep 17 18:52:28.984548 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:28.984530    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:52:29.364948 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:29.364889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" event={"ID":"2b2e77c1-b92a-4f14-8b74-623649b97e79","Type":"ContainerStarted","Data":"2a33f717500a560ad3fc3f63d7a324c21a453c8e1af31745971bc45963ad22e8"}
Sep 17 18:52:32.380948 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:32.380911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" event={"ID":"2b2e77c1-b92a-4f14-8b74-623649b97e79","Type":"ContainerStarted","Data":"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"}
Sep 17 18:52:32.381438 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:32.381022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:52:32.398188 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:32.398140    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podStartSLOduration=1.519981357 podStartE2EDuration="4.398125078s" totalImagesPullingTime="2.878143721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:32.397475061 +0000 UTC m=+769.538901287" watchObservedRunningTime="2026-09-17 18:52:32.398125078 +0000 UTC m=+769.539551290"
Sep 17 18:52:36.889193 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.889158    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:52:36.892219 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.892194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.894865 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.894837    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 18:52:36.895746 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.895706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-gtw6r\""
Sep 17 18:52:36.895880 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.895802    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 18:52:36.895880 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.895873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 18:52:36.905361 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.905338    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:52:36.988395 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.988395 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.988602 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988420    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.988602 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.988602 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:36.988602 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:36.988556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpngs\" (UniqueName: \"kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090009 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.089981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090183 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090183 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090183 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090183 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090435 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpngs\" (UniqueName: \"kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090489 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090489 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.090586 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.090520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.092563 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.092546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.092853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.092832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.101972 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.101946    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpngs\" (UniqueName: \"kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs\") pod \"fc-smoke-test-kserve-7f4687897-vsg8x\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.206431 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.206351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:37.486751 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:37.486728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:52:37.490129 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:52:37.490105    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761fb02a_cb74_4997_a71e_4afd02af820d.slice/crio-d357ca4b5e320daaaa53e821014bce46b2c11612aae0242338cef4b455aa3448 WatchSource:0}: Error finding container d357ca4b5e320daaaa53e821014bce46b2c11612aae0242338cef4b455aa3448: Status 404 returned error can't find the container with id d357ca4b5e320daaaa53e821014bce46b2c11612aae0242338cef4b455aa3448
Sep 17 18:52:38.412455 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:38.412423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" event={"ID":"761fb02a-cb74-4997-a71e-4afd02af820d","Type":"ContainerStarted","Data":"d357ca4b5e320daaaa53e821014bce46b2c11612aae0242338cef4b455aa3448"}
Sep 17 18:52:39.418458 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.418419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" event={"ID":"761fb02a-cb74-4997-a71e-4afd02af820d","Type":"ContainerStarted","Data":"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"}
Sep 17 18:52:39.419949 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.419926    2561 generic.go:356] "Generic (PLEG): container finished" podID="5a413b68-67ab-4c5f-a450-cd459e093103" containerID="21d7e826dc312fb934adc28289990ce28ec3752500e3ab92370cd42ba0aa6c23" exitCode=0
Sep 17 18:52:39.420067 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.419966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" event={"ID":"5a413b68-67ab-4c5f-a450-cd459e093103","Type":"ContainerDied","Data":"21d7e826dc312fb934adc28289990ce28ec3752500e3ab92370cd42ba0aa6c23"}
Sep 17 18:52:39.420277 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.420264    2561 scope.go:117] "RemoveContainer" containerID="21d7e826dc312fb934adc28289990ce28ec3752500e3ab92370cd42ba0aa6c23"
Sep 17 18:52:39.421170 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.421137    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:52:39.421243 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.421227    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:52:39.421280 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.421267    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:52:39.441517 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:39.441460    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" podStartSLOduration=1.950654428 podStartE2EDuration="3.441440682s" totalImagesPullingTime="1.490786254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:39.440957136 +0000 UTC m=+776.582383354" watchObservedRunningTime="2026-09-17 18:52:39.441440682 +0000 UTC m=+776.582866894"
Sep 17 18:52:40.206704 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:40.206666    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:40.429562 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:40.428778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" event={"ID":"5a413b68-67ab-4c5f-a450-cd459e093103","Type":"ContainerStarted","Data":"5a5162e9b1d9f542cab59ddd293c8cd93e61ae34b4781bfa0e0721f1300f7194"}
Sep 17 18:52:40.823433 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:40.823399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:52:41.038743 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:41.038710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:52:41.039114 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:41.039090    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused" start-of-body=
Sep 17 18:52:41.039289 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:41.039265    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podUID="5a413b68-67ab-4c5f-a450-cd459e093103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused"
Sep 17 18:52:42.037835 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:42.037802    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused" start-of-body=
Sep 17 18:52:42.038430 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:42.038401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podUID="5a413b68-67ab-4c5f-a450-cd459e093103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused"
Sep 17 18:52:43.037862 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:43.037826    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused" start-of-body=
Sep 17 18:52:43.038474 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:43.038441    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podUID="5a413b68-67ab-4c5f-a450-cd459e093103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused"
Sep 17 18:52:44.038458 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:44.038422    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused" start-of-body=
Sep 17 18:52:44.039073 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:44.039032    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podUID="5a413b68-67ab-4c5f-a450-cd459e093103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused"
Sep 17 18:52:45.038530 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:45.038492    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused" start-of-body=
Sep 17 18:52:45.039182 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:45.039149    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k" podUID="5a413b68-67ab-4c5f-a450-cd459e093103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.17:15021/healthz/ready\": dial tcp 10.134.0.17:15021: connect: connection refused"
Sep 17 18:52:46.056460 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:46.056423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:52:46.057535 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:46.057514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k"
Sep 17 18:52:47.207496 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:47.207459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:47.220282 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:47.220257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:52:47.476063 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:52:47.475978    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:53:03.387215 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:03.387175    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:10.801165 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:10.801134    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:53:10.802774 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:10.802735    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:18.044491 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.044459    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:53:18.054290 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:53:18.054253    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:53:18.054290 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:18.054290 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:18.054290 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295" cmd=["/bin/sleep","15"]
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.054309    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" message="" err=<
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:53:18.054324    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" containerName="main" containerID="cri-o://00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"
Sep 17 18:53:18.054541 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.054372    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" containerName="main" containerID="cri-o://00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295" gracePeriod=60
Sep 17 18:53:18.437329 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.437299    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:53:18.473018 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.472985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473181 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473181 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpngs\" (UniqueName: \"kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473181 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473350 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473231    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473350 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home\") pod \"761fb02a-cb74-4997-a71e-4afd02af820d\" (UID: \"761fb02a-cb74-4997-a71e-4afd02af820d\") "
Sep 17 18:53:18.473350 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache" (OuterVolumeSpecName: "model-cache") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:18.473505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473439    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:18.473565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473542    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.473565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473561    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.473681 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.473652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home" (OuterVolumeSpecName: "home") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:18.475562 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.475534    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm" (OuterVolumeSpecName: "dshm") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:18.475562 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.475548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs" (OuterVolumeSpecName: "kube-api-access-rpngs") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "kube-api-access-rpngs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:18.475755 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.475728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "761fb02a-cb74-4997-a71e-4afd02af820d" (UID: "761fb02a-cb74-4997-a71e-4afd02af820d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:18.574036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.573993    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.574036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.574021    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761fb02a-cb74-4997-a71e-4afd02af820d-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.574036 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.574041    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpngs\" (UniqueName: \"kubernetes.io/projected/761fb02a-cb74-4997-a71e-4afd02af820d-kube-api-access-rpngs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.574244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.574050    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761fb02a-cb74-4997-a71e-4afd02af820d-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:18.606746 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.606675    2561 generic.go:356] "Generic (PLEG): container finished" podID="761fb02a-cb74-4997-a71e-4afd02af820d" containerID="00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295" exitCode=0
Sep 17 18:53:18.606746 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.606720    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"
Sep 17 18:53:18.606937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.606722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" event={"ID":"761fb02a-cb74-4997-a71e-4afd02af820d","Type":"ContainerDied","Data":"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"}
Sep 17 18:53:18.606937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.606846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x" event={"ID":"761fb02a-cb74-4997-a71e-4afd02af820d","Type":"ContainerDied","Data":"d357ca4b5e320daaaa53e821014bce46b2c11612aae0242338cef4b455aa3448"}
Sep 17 18:53:18.606937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.606876    2561 scope.go:117] "RemoveContainer" containerID="00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"
Sep 17 18:53:18.615719 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.615703    2561 scope.go:117] "RemoveContainer" containerID="00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"
Sep 17 18:53:18.616099 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:53:18.616082    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295\": container with ID starting with 00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295 not found: ID does not exist" containerID="00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"
Sep 17 18:53:18.616149 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.616106    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295"} err="failed to get container status \"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295\": rpc error: code = NotFound desc = could not find container \"00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295\": container with ID starting with 00d226b3735b3fc76496c4d1d7620b2c7060c5ca733dc309f5cc9ac890b29295 not found: ID does not exist"
Sep 17 18:53:18.627753 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.627721    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:53:18.635384 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:18.635360    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vsg8x"]
Sep 17 18:53:19.086623 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:19.086593    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" path="/var/lib/kubelet/pods/761fb02a-cb74-4997-a71e-4afd02af820d/volumes"
Sep 17 18:53:20.803310 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:20.803270    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:25.822321 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:25.822271    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" containerID="cri-o://d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4" gracePeriod=45
Sep 17 18:53:30.803468 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:30.803431    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:40.803046 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:40.803008    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:50.803339 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:50.803303    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:56.065064 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.065039    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:53:56.104836 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.104813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs\") pod \"2b2e77c1-b92a-4f14-8b74-623649b97e79\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") "
Sep 17 18:53:56.104970 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.104864    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbzhc\" (UniqueName: \"kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc\") pod \"2b2e77c1-b92a-4f14-8b74-623649b97e79\" (UID: \"2b2e77c1-b92a-4f14-8b74-623649b97e79\") "
Sep 17 18:53:56.107151 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.107117    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b2e77c1-b92a-4f14-8b74-623649b97e79" (UID: "2b2e77c1-b92a-4f14-8b74-623649b97e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:56.107248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.107187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc" (OuterVolumeSpecName: "kube-api-access-gbzhc") pod "2b2e77c1-b92a-4f14-8b74-623649b97e79" (UID: "2b2e77c1-b92a-4f14-8b74-623649b97e79"). InnerVolumeSpecName "kube-api-access-gbzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:56.206192 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.206162    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2e77c1-b92a-4f14-8b74-623649b97e79-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:56.206192 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.206186    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzhc\" (UniqueName: \"kubernetes.io/projected/2b2e77c1-b92a-4f14-8b74-623649b97e79-kube-api-access-gbzhc\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:53:56.775443 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.775405    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerID="d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4" exitCode=0
Sep 17 18:53:56.775612 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.775455    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"
Sep 17 18:53:56.775612 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.775492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" event={"ID":"2b2e77c1-b92a-4f14-8b74-623649b97e79","Type":"ContainerDied","Data":"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"}
Sep 17 18:53:56.775612 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.775547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6" event={"ID":"2b2e77c1-b92a-4f14-8b74-623649b97e79","Type":"ContainerDied","Data":"2a33f717500a560ad3fc3f63d7a324c21a453c8e1af31745971bc45963ad22e8"}
Sep 17 18:53:56.775612 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.775571    2561 scope.go:117] "RemoveContainer" containerID="d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"
Sep 17 18:53:56.785227 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.785204    2561 scope.go:117] "RemoveContainer" containerID="d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"
Sep 17 18:53:56.790611 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:53:56.790586    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4\": container with ID starting with d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4 not found: ID does not exist" containerID="d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"
Sep 17 18:53:56.790722 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.790621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4"} err="failed to get container status \"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4\": rpc error: code = NotFound desc = could not find container \"d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4\": container with ID starting with d4fee64c9b71131fbf2dac48244939ebef441624b53903d77de720136a4d8de4 not found: ID does not exist"
Sep 17 18:53:56.796575 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.796554    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:53:56.800921 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:56.800881    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5c4dt6"]
Sep 17 18:53:57.086391 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:53:57.086362    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" path="/var/lib/kubelet/pods/2b2e77c1-b92a-4f14-8b74-623649b97e79/volumes"
Sep 17 18:54:12.149977 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.149932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150332    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main"
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main"
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150390    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" containerName="main"
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150396    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" containerName="main"
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150464    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2e77c1-b92a-4f14-8b74-623649b97e79" containerName="main"
Sep 17 18:54:12.150486 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.150477    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="761fb02a-cb74-4997-a71e-4afd02af820d" containerName="main"
Sep 17 18:54:12.153551 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.153531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.156979 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.156952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-6cxdv\""
Sep 17 18:54:12.157096 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.157026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-25p5g\""
Sep 17 18:54:12.157096 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.157060    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:54:12.157213 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.157032    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:54:12.157334 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.157314    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:54:12.166161 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.166141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:54:12.248581 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.248548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2s7\" (UniqueName: \"kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.248729 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.248602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.248729 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.248710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.248886 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.248744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.349722 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.349693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.349722 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.349728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.349981 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.349762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2s7\" (UniqueName: \"kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.349981 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.349798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.350194 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.350172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.350264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.350228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.352238 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.352222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.363993 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.363970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2s7\" (UniqueName: \"kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.464570 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.464500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.741654 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.741575    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:54:12.744825 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:54:12.744793    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377154ab_df08_4395_8a44_5ec605b821ce.slice/crio-963410acdbde454c23133296f804f3fab9a077520683cb229278e64026bcb753 WatchSource:0}: Error finding container 963410acdbde454c23133296f804f3fab9a077520683cb229278e64026bcb753: Status 404 returned error can't find the container with id 963410acdbde454c23133296f804f3fab9a077520683cb229278e64026bcb753
Sep 17 18:54:12.848472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.848258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" event={"ID":"377154ab-df08-4395-8a44-5ec605b821ce","Type":"ContainerStarted","Data":"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"}
Sep 17 18:54:12.848472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.848328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" event={"ID":"377154ab-df08-4395-8a44-5ec605b821ce","Type":"ContainerStarted","Data":"963410acdbde454c23133296f804f3fab9a077520683cb229278e64026bcb753"}
Sep 17 18:54:12.848472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.848360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:12.871204 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:12.871078    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podStartSLOduration=0.871057467 podStartE2EDuration="871.057467ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:12.869468944 +0000 UTC m=+870.010895161" watchObservedRunningTime="2026-09-17 18:54:12.871057467 +0000 UTC m=+870.012483679"
Sep 17 18:54:38.895773 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.895738    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:54:38.898908 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.898874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.902411 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.902389    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:54:38.903157 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.903140    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:54:38.903355 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.903341    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:54:38.903428 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.903411    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-zvl2n\""
Sep 17 18:54:38.912437 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.912416    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:54:38.985165 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.985282 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.985282 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z68v\" (UniqueName: \"kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.985382 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.985382 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:38.985382 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:38.985363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086349 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z68v\" (UniqueName: \"kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086463 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086463 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086463 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086463 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086667 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086867 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.086867 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.087070 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.086884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.088944 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.088925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.089201 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.089184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.100871 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.100846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z68v\" (UniqueName: \"kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v\") pod \"fc-concurrency-test-kserve-b5d44467c-dqqlp\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.209285 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.209207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:39.498786 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.498760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:54:39.500729 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:54:39.500707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794020ab_8734_4df0_ad85_9d4435ae5321.slice/crio-e3fc785f0d38400e954026c5ffc449f9b4e3c66dd13c090d32c6aab5f259b0ae WatchSource:0}: Error finding container e3fc785f0d38400e954026c5ffc449f9b4e3c66dd13c090d32c6aab5f259b0ae: Status 404 returned error can't find the container with id e3fc785f0d38400e954026c5ffc449f9b4e3c66dd13c090d32c6aab5f259b0ae
Sep 17 18:54:39.975812 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.975764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" event={"ID":"794020ab-8734-4df0-ad85-9d4435ae5321","Type":"ContainerStarted","Data":"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"}
Sep 17 18:54:39.976214 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:39.975845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" event={"ID":"794020ab-8734-4df0-ad85-9d4435ae5321","Type":"ContainerStarted","Data":"e3fc785f0d38400e954026c5ffc449f9b4e3c66dd13c090d32c6aab5f259b0ae"}
Sep 17 18:54:40.003341 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:40.003280    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" podStartSLOduration=2.003262714 podStartE2EDuration="2.003262714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:40.002588673 +0000 UTC m=+897.144014885" watchObservedRunningTime="2026-09-17 18:54:40.003262714 +0000 UTC m=+897.144688926"
Sep 17 18:54:40.209540 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:40.209509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:43.856939 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:43.856889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:54:44.991793 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:44.991765    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:54:45.025120 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:45.025100    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:54:49.210158 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:49.210119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:49.222725 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:49.222699    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:50.034091 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:50.034061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:54:53.112369 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:53.112336    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:54:53.856575 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:54:53.856532    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:03.857035 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:03.856997    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:08.135074 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:08.135031    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" containerID="cri-o://be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2" gracePeriod=45
Sep 17 18:55:13.856614 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:13.856580    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:13.857118 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:13.856673    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:55:18.328676 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.328644    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:55:18.339023 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:55:18.338967    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:55:18.339023 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:55:18.339023 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:55:18.339023 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503" cmd=["/bin/sleep","15"]
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.339042    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" message="" err=<
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:55:18.339054    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" containerName="main" containerID="cri-o://05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"
Sep 17 18:55:18.339321 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.339107    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" containerName="main" containerID="cri-o://05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503" gracePeriod=60
Sep 17 18:55:18.682878 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.682828    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:55:18.724027 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z68v\" (UniqueName: \"kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724203 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm\") pod \"794020ab-8734-4df0-ad85-9d4435ae5321\" (UID: \"794020ab-8734-4df0-ad85-9d4435ae5321\") "
Sep 17 18:55:18.724472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724288    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home" (OuterVolumeSpecName: "home") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:18.724472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724414    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:18.724472 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724426    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:18.724666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.724471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache" (OuterVolumeSpecName: "model-cache") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:18.726905 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.726864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v" (OuterVolumeSpecName: "kube-api-access-2z68v") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "kube-api-access-2z68v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:18.727055 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.727035    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm" (OuterVolumeSpecName: "dshm") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:18.727137 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.727110    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "794020ab-8734-4df0-ad85-9d4435ae5321" (UID: "794020ab-8734-4df0-ad85-9d4435ae5321"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:55:18.825671 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.825644    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:18.825671 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.825669    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z68v\" (UniqueName: \"kubernetes.io/projected/794020ab-8734-4df0-ad85-9d4435ae5321-kube-api-access-2z68v\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:18.825671 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.825680    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:18.825917 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.825688    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794020ab-8734-4df0-ad85-9d4435ae5321-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:18.825917 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:18.825696    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794020ab-8734-4df0-ad85-9d4435ae5321-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:19.156312 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.156274    2561 generic.go:356] "Generic (PLEG): container finished" podID="794020ab-8734-4df0-ad85-9d4435ae5321" containerID="05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503" exitCode=0
Sep 17 18:55:19.156468 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.156340    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"
Sep 17 18:55:19.156468 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.156351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" event={"ID":"794020ab-8734-4df0-ad85-9d4435ae5321","Type":"ContainerDied","Data":"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"}
Sep 17 18:55:19.156468 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.156430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp" event={"ID":"794020ab-8734-4df0-ad85-9d4435ae5321","Type":"ContainerDied","Data":"e3fc785f0d38400e954026c5ffc449f9b4e3c66dd13c090d32c6aab5f259b0ae"}
Sep 17 18:55:19.156468 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.156451    2561 scope.go:117] "RemoveContainer" containerID="05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"
Sep 17 18:55:19.165198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.165183    2561 scope.go:117] "RemoveContainer" containerID="05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"
Sep 17 18:55:19.165465 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:55:19.165448    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503\": container with ID starting with 05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503 not found: ID does not exist" containerID="05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"
Sep 17 18:55:19.165523 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.165471    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503"} err="failed to get container status \"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503\": rpc error: code = NotFound desc = could not find container \"05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503\": container with ID starting with 05f724aa227de2fbafd91abd834eded886409fb9eb2c3eef2b51e4c843f2f503 not found: ID does not exist"
Sep 17 18:55:19.186105 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.186068    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:55:19.187417 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:19.187397    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-dqqlp"]
Sep 17 18:55:21.086496 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:21.086464    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" path="/var/lib/kubelet/pods/794020ab-8734-4df0-ad85-9d4435ae5321/volumes"
Sep 17 18:55:23.856566 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:23.856531    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:33.856361 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:33.856322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:55:38.387710 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.387685    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:55:38.489443 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489373    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache\") pod \"377154ab-df08-4395-8a44-5ec605b821ce\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") "
Sep 17 18:55:38.489443 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr2s7\" (UniqueName: \"kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7\") pod \"377154ab-df08-4395-8a44-5ec605b821ce\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") "
Sep 17 18:55:38.489663 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache\") pod \"377154ab-df08-4395-8a44-5ec605b821ce\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") "
Sep 17 18:55:38.489663 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489508    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs\") pod \"377154ab-df08-4395-8a44-5ec605b821ce\" (UID: \"377154ab-df08-4395-8a44-5ec605b821ce\") "
Sep 17 18:55:38.489771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "377154ab-df08-4395-8a44-5ec605b821ce" (UID: "377154ab-df08-4395-8a44-5ec605b821ce"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:38.489771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "377154ab-df08-4395-8a44-5ec605b821ce" (UID: "377154ab-df08-4395-8a44-5ec605b821ce"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:55:38.489871 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489801    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-tokenizer-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:38.489871 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.489813    2561 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/377154ab-df08-4395-8a44-5ec605b821ce-cachi2-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:38.491937 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.491912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "377154ab-df08-4395-8a44-5ec605b821ce" (UID: "377154ab-df08-4395-8a44-5ec605b821ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:55:38.492043 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.491966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7" (OuterVolumeSpecName: "kube-api-access-rr2s7") pod "377154ab-df08-4395-8a44-5ec605b821ce" (UID: "377154ab-df08-4395-8a44-5ec605b821ce"). InnerVolumeSpecName "kube-api-access-rr2s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:55:38.590717 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.590686    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2s7\" (UniqueName: \"kubernetes.io/projected/377154ab-df08-4395-8a44-5ec605b821ce-kube-api-access-rr2s7\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:38.590717 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:38.590715    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377154ab-df08-4395-8a44-5ec605b821ce-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:55:39.243770 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.243736    2561 generic.go:356] "Generic (PLEG): container finished" podID="377154ab-df08-4395-8a44-5ec605b821ce" containerID="be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2" exitCode=0
Sep 17 18:55:39.243954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.243781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" event={"ID":"377154ab-df08-4395-8a44-5ec605b821ce","Type":"ContainerDied","Data":"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"}
Sep 17 18:55:39.243954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.243787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"
Sep 17 18:55:39.243954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.243821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq" event={"ID":"377154ab-df08-4395-8a44-5ec605b821ce","Type":"ContainerDied","Data":"963410acdbde454c23133296f804f3fab9a077520683cb229278e64026bcb753"}
Sep 17 18:55:39.243954 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.243846    2561 scope.go:117] "RemoveContainer" containerID="be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"
Sep 17 18:55:39.252651 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.252485    2561 scope.go:117] "RemoveContainer" containerID="be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"
Sep 17 18:55:39.252923 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:55:39.252885    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2\": container with ID starting with be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2 not found: ID does not exist" containerID="be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"
Sep 17 18:55:39.253019 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.252934    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2"} err="failed to get container status \"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2\": rpc error: code = NotFound desc = could not find container \"be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2\": container with ID starting with be9204af79e0902f1898274937ca33339da1448f2fbf6ec27167ce9d76a32da2 not found: ID does not exist"
Sep 17 18:55:39.263796 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.263775    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:55:39.269844 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:39.269823    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcpgzwq"]
Sep 17 18:55:41.086666 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:41.086631    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377154ab-df08-4395-8a44-5ec605b821ce" path="/var/lib/kubelet/pods/377154ab-df08-4395-8a44-5ec605b821ce/volumes"
Sep 17 18:55:48.792265 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792228    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:55:48.792746 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792699    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" containerName="main"
Sep 17 18:55:48.792746 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792717    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" containerName="main"
Sep 17 18:55:48.792863 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792759    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main"
Sep 17 18:55:48.792863 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792768    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main"
Sep 17 18:55:48.792863 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792842    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="377154ab-df08-4395-8a44-5ec605b821ce" containerName="main"
Sep 17 18:55:48.792863 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.792855    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="794020ab-8734-4df0-ad85-9d4435ae5321" containerName="main"
Sep 17 18:55:48.797730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.797709    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.802312 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.802286    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:55:48.802312 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.802286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:55:48.802496 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.802380    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:55:48.802931 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.802916    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6rxm2\""
Sep 17 18:55:48.815244 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.815224    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:55:48.877686 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.877815 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.877815 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpvw\" (UniqueName: \"kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.877815 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.877815 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.877981 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.877879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.978751 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.978693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.978885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.978778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.978885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.978812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.978885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.978843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.979032 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.978885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpvw\" (UniqueName: \"kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.979128 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.979033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.979212 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.979191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.979268 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.979243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.979386 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.979365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.981136 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.981120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.981519 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.981498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:48.989104 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:48.989084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpvw\" (UniqueName: \"kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:49.107297 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:49.107267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:49.388807 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:49.388783    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:55:49.390739 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:55:49.390710    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91770344_218b_405e_acc0_f371882355f2.slice/crio-b9843a73a99613bb5d615dc95fafc432bf7a38c373fc17e5217a52c00973148b WatchSource:0}: Error finding container b9843a73a99613bb5d615dc95fafc432bf7a38c373fc17e5217a52c00973148b: Status 404 returned error can't find the container with id b9843a73a99613bb5d615dc95fafc432bf7a38c373fc17e5217a52c00973148b
Sep 17 18:55:50.291970 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:50.291938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" event={"ID":"91770344-218b-405e-acc0-f371882355f2","Type":"ContainerStarted","Data":"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"}
Sep 17 18:55:50.292343 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:50.291995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" event={"ID":"91770344-218b-405e-acc0-f371882355f2","Type":"ContainerStarted","Data":"b9843a73a99613bb5d615dc95fafc432bf7a38c373fc17e5217a52c00973148b"}
Sep 17 18:55:50.316861 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:50.316818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" podStartSLOduration=2.316799758 podStartE2EDuration="2.316799758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:50.315666698 +0000 UTC m=+967.457092910" watchObservedRunningTime="2026-09-17 18:55:50.316799758 +0000 UTC m=+967.458225951"
Sep 17 18:55:51.108275 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:51.108244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:59.107635 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:59.107597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:59.120030 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:59.120007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:55:59.342223 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:55:59.342192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:56:08.803052 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.803016    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:08.806399 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.806355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.812381 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.812357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 18:56:08.813497 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.813461    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:56:08.813597 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.813518    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 18:56:08.820553 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.820532    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-c5q5q\""
Sep 17 18:56:08.848324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848324 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslnl\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848521 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848721 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848721 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.848721 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.848616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.863328 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.863298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:08.949885 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.949854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950085 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.949920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslnl\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950085 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.949976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950085 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950238 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950238 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950238 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950238 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950434 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950607 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950942 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.950942 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.950738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.951339 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.951312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.952853 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.952829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.953145 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.953127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.960255 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.960234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:08.960616 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:08.960597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslnl\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl\") pod \"router-gateway-1-openshift-default-6b46477c5b-vr2bw\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:09.117136 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:09.117109    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:09.426175 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:09.426147    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:09.428636 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:56:09.428598    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58596f07_7f21_46b6_b5e4_f8703e4208f8.slice/crio-bf33885016d8e892a894237515415a3e62409b862d904d63c36ef6d37c397117 WatchSource:0}: Error finding container bf33885016d8e892a894237515415a3e62409b862d904d63c36ef6d37c397117: Status 404 returned error can't find the container with id bf33885016d8e892a894237515415a3e62409b862d904d63c36ef6d37c397117
Sep 17 18:56:09.431159 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:09.431130    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:09.431248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:09.431194    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:09.431248 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:09.431221    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:10.379663 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:10.379616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" event={"ID":"58596f07-7f21-46b6-b5e4-f8703e4208f8","Type":"ContainerStarted","Data":"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"}
Sep 17 18:56:10.380125 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:10.379681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" event={"ID":"58596f07-7f21-46b6-b5e4-f8703e4208f8","Type":"ContainerStarted","Data":"bf33885016d8e892a894237515415a3e62409b862d904d63c36ef6d37c397117"}
Sep 17 18:56:10.403694 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:10.403647    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" podStartSLOduration=2.403630105 podStartE2EDuration="2.403630105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:10.402566632 +0000 UTC m=+987.543992855" watchObservedRunningTime="2026-09-17 18:56:10.403630105 +0000 UTC m=+987.545056317"
Sep 17 18:56:11.117496 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:11.117465    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:11.122599 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:11.122573    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:11.384098 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:11.384017    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:11.384966 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:11.384947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:23.914089 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.914057    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:23.917437 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.917415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.921773 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.921728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-7dmgz\""
Sep 17 18:56:23.922186 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.922137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:56:23.922317 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.922293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:56:23.922632 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.922618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:56:23.940919 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.934661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:23.984279 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984418 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984418 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984418 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984524 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984618 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:23.984618 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:23.984555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ln74\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085098 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085284 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085649 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ln74\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085706 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085706 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085796 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085846 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.085951 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.085931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.087861 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.087839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.087982 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.087868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.093214 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.093190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ln74\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.093988 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.093968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-jnlmx\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.249011 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.248642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:24.530196 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.530072    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:24.538998 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.538958    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:24.539196 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.539171    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:24.539321 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:24.539305    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:56:25.454326 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:25.454291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" event={"ID":"b1999d77-165e-40d3-b3e7-8d36b0f77045","Type":"ContainerStarted","Data":"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"}
Sep 17 18:56:25.454842 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:25.454350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" event={"ID":"b1999d77-165e-40d3-b3e7-8d36b0f77045","Type":"ContainerStarted","Data":"dd86965bfef04f1e6febd375f60e3923bc2e763370cfbcaad6b83dbacef6b620"}
Sep 17 18:56:25.476665 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:25.476603    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" podStartSLOduration=2.476583755 podStartE2EDuration="2.476583755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:25.476324747 +0000 UTC m=+1002.617750960" watchObservedRunningTime="2026-09-17 18:56:25.476583755 +0000 UTC m=+1002.618009965"
Sep 17 18:56:26.249229 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:26.249187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:26.254515 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:26.254488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:26.459429 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:26.459395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:26.460482 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:26.460461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:28.622198 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:28.622161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:28.622664 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:28.622408    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" podUID="58596f07-7f21-46b6-b5e4-f8703e4208f8" containerName="istio-proxy" containerID="cri-o://e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825" gracePeriod=30
Sep 17 18:56:31.462757 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.462729    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:31.483697 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.483668    2561 generic.go:356] "Generic (PLEG): container finished" podID="58596f07-7f21-46b6-b5e4-f8703e4208f8" containerID="e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825" exitCode=0
Sep 17 18:56:31.483810 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.483718    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"
Sep 17 18:56:31.483810 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.483764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" event={"ID":"58596f07-7f21-46b6-b5e4-f8703e4208f8","Type":"ContainerDied","Data":"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"}
Sep 17 18:56:31.483880 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.483829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw" event={"ID":"58596f07-7f21-46b6-b5e4-f8703e4208f8","Type":"ContainerDied","Data":"bf33885016d8e892a894237515415a3e62409b862d904d63c36ef6d37c397117"}
Sep 17 18:56:31.483880 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.483861    2561 scope.go:117] "RemoveContainer" containerID="e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"
Sep 17 18:56:31.493644 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.493624    2561 scope.go:117] "RemoveContainer" containerID="e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"
Sep 17 18:56:31.494018 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:56:31.493991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825\": container with ID starting with e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825 not found: ID does not exist" containerID="e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"
Sep 17 18:56:31.494095 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.494031    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825"} err="failed to get container status \"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825\": rpc error: code = NotFound desc = could not find container \"e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825\": container with ID starting with e25c1e0bacc74756ed9f7945b1bc5b75e15d3c964cb97c7e8e901cdeda873825 not found: ID does not exist"
Sep 17 18:56:31.555493 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555652 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555508    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555652 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555652 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555652 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555652 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555928 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555928 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rslnl\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555928 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo\") pod \"58596f07-7f21-46b6-b5e4-f8703e4208f8\" (UID: \"58596f07-7f21-46b6-b5e4-f8703e4208f8\") "
Sep 17 18:56:31.555928 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.555742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:31.556135 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.556035    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-data\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.556135 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.556032    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:31.556135 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.556084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:31.556290 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.556114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:31.556290 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.556162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:31.558257 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.558210    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:31.558385 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.558288    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:31.558385 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.558358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl" (OuterVolumeSpecName: "kube-api-access-rslnl") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "kube-api-access-rslnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:31.558676 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.558652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58596f07-7f21-46b6-b5e4-f8703e4208f8" (UID: "58596f07-7f21-46b6-b5e4-f8703e4208f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:56:31.656527 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656488    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656527 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656519    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-envoy\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656527 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656532    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-workload-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656759 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656541    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58596f07-7f21-46b6-b5e4-f8703e4208f8-credential-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656759 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656550    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rslnl\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-kube-api-access-rslnl\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656759 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656559    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-podinfo\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656759 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656567    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58596f07-7f21-46b6-b5e4-f8703e4208f8-istiod-ca-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.656759 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.656575    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58596f07-7f21-46b6-b5e4-f8703e4208f8-istio-token\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:31.806234 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.806205    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:31.810242 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:31.810221    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-vr2bw"]
Sep 17 18:56:33.086208 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:33.086173    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58596f07-7f21-46b6-b5e4-f8703e4208f8" path="/var/lib/kubelet/pods/58596f07-7f21-46b6-b5e4-f8703e4208f8/volumes"
Sep 17 18:56:47.488709 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:47.488669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:47.489187 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:47.488932    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" podUID="b1999d77-165e-40d3-b3e7-8d36b0f77045" containerName="istio-proxy" containerID="cri-o://14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507" gracePeriod=30
Sep 17 18:56:48.833569 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.833545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:48.923611 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923582    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923628    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923671    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923719    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923769 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ln74\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923996 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.923996 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923918    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:48.923996 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:48.923996 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.923955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.924192 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.924192 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy\") pod \"b1999d77-165e-40d3-b3e7-8d36b0f77045\" (UID: \"b1999d77-165e-40d3-b3e7-8d36b0f77045\") "
Sep 17 18:56:48.924295 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:48.924295 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:48.924389 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:48.924526 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924486    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-credential-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:48.924526 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924510    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-data\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:48.924631 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924525    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1999d77-165e-40d3-b3e7-8d36b0f77045-istiod-ca-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:48.924631 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924538    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:48.924631 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.924552    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-workload-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:48.926287 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.926259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:48.926470 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.926450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74" (OuterVolumeSpecName: "kube-api-access-8ln74") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "kube-api-access-8ln74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:48.926712 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.926686    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:56:48.926712 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:48.926702    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1999d77-165e-40d3-b3e7-8d36b0f77045" (UID: "b1999d77-165e-40d3-b3e7-8d36b0f77045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:49.025232 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.025208    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-envoy\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:49.025232 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.025231    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-token\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:49.025388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.025241    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ln74\" (UniqueName: \"kubernetes.io/projected/b1999d77-165e-40d3-b3e7-8d36b0f77045-kube-api-access-8ln74\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:49.025388 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.025250    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1999d77-165e-40d3-b3e7-8d36b0f77045-istio-podinfo\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:56:49.578469 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.578431    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1999d77-165e-40d3-b3e7-8d36b0f77045" containerID="14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507" exitCode=0
Sep 17 18:56:49.578653 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.578474    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"
Sep 17 18:56:49.578653 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.578516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" event={"ID":"b1999d77-165e-40d3-b3e7-8d36b0f77045","Type":"ContainerDied","Data":"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"}
Sep 17 18:56:49.578653 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.578565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx" event={"ID":"b1999d77-165e-40d3-b3e7-8d36b0f77045","Type":"ContainerDied","Data":"dd86965bfef04f1e6febd375f60e3923bc2e763370cfbcaad6b83dbacef6b620"}
Sep 17 18:56:49.578653 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.578595    2561 scope.go:117] "RemoveContainer" containerID="14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"
Sep 17 18:56:49.587178 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.587149    2561 scope.go:117] "RemoveContainer" containerID="14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"
Sep 17 18:56:49.587446 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:56:49.587426    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507\": container with ID starting with 14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507 not found: ID does not exist" containerID="14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"
Sep 17 18:56:49.587505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.587454    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507"} err="failed to get container status \"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507\": rpc error: code = NotFound desc = could not find container \"14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507\": container with ID starting with 14b3933059a57f467a85f2921d9f08ca45c4f59baa9ad6d24a181495e1bec507 not found: ID does not exist"
Sep 17 18:56:49.600762 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.600741    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:49.613484 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:49.613463    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-jnlmx"]
Sep 17 18:56:51.086832 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:56:51.086800    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1999d77-165e-40d3-b3e7-8d36b0f77045" path="/var/lib/kubelet/pods/b1999d77-165e-40d3-b3e7-8d36b0f77045/volumes"
Sep 17 18:57:36.636399 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:36.636322    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:57:36.645267 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:57:36.645231    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:57:36.645267 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.645267 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.645267 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10" cmd=["/bin/sleep","15"]
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:36.645289    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" message="" err=<
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:57:36.645303    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" podUID="91770344-218b-405e-acc0-f371882355f2" containerName="main" containerID="cri-o://6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"
Sep 17 18:57:36.645505 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:36.645351    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" podUID="91770344-218b-405e-acc0-f371882355f2" containerName="main" containerID="cri-o://6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10" gracePeriod=60
Sep 17 18:57:36.989042 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:36.989017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:57:37.031808 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.031782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.031808 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.031813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.031867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.031910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vpvw\" (UniqueName: \"kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.031967    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm\") pod \"91770344-218b-405e-acc0-f371882355f2\" (UID: \"91770344-218b-405e-acc0-f371882355f2\") "
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032117    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.032205 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home" (OuterVolumeSpecName: "home") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.032596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.032596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032356    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.032596 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.032371    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.034458 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.034433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:37.034579 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.034557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm" (OuterVolumeSpecName: "dshm") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.034832 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.034808    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw" (OuterVolumeSpecName: "kube-api-access-2vpvw") pod "91770344-218b-405e-acc0-f371882355f2" (UID: "91770344-218b-405e-acc0-f371882355f2"). InnerVolumeSpecName "kube-api-access-2vpvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:37.133123 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.133094    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.133123 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.133117    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91770344-218b-405e-acc0-f371882355f2-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.133123 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.133127    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vpvw\" (UniqueName: \"kubernetes.io/projected/91770344-218b-405e-acc0-f371882355f2-kube-api-access-2vpvw\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.133305 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.133135    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91770344-218b-405e-acc0-f371882355f2-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.796609 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.796573    2561 generic.go:356] "Generic (PLEG): container finished" podID="91770344-218b-405e-acc0-f371882355f2" containerID="6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10" exitCode=0
Sep 17 18:57:37.797001 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.796625    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"
Sep 17 18:57:37.797001 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.796661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" event={"ID":"91770344-218b-405e-acc0-f371882355f2","Type":"ContainerDied","Data":"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"}
Sep 17 18:57:37.797001 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.796718    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt" event={"ID":"91770344-218b-405e-acc0-f371882355f2","Type":"ContainerDied","Data":"b9843a73a99613bb5d615dc95fafc432bf7a38c373fc17e5217a52c00973148b"}
Sep 17 18:57:37.797001 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.796743    2561 scope.go:117] "RemoveContainer" containerID="6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"
Sep 17 18:57:37.805115 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.805097    2561 scope.go:117] "RemoveContainer" containerID="6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"
Sep 17 18:57:37.805370 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:57:37.805350    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10\": container with ID starting with 6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10 not found: ID does not exist" containerID="6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"
Sep 17 18:57:37.805430 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.805379    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10"} err="failed to get container status \"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10\": rpc error: code = NotFound desc = could not find container \"6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10\": container with ID starting with 6daccbb9f872490a1e7e36c046f703d0b12eda712aa4e50cb75fd3a987584d10 not found: ID does not exist"
Sep 17 18:57:37.815339 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.815316    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:57:37.821171 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:37.821147    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-dxbkt"]
Sep 17 18:57:39.086028 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:57:39.085996    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91770344-218b-405e-acc0-f371882355f2" path="/var/lib/kubelet/pods/91770344-218b-405e-acc0-f371882355f2/volumes"
Sep 17 18:58:45.938267 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938237    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938544    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91770344-218b-405e-acc0-f371882355f2" containerName="main"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938556    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91770344-218b-405e-acc0-f371882355f2" containerName="main"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938564    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1999d77-165e-40d3-b3e7-8d36b0f77045" containerName="istio-proxy"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938569    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1999d77-165e-40d3-b3e7-8d36b0f77045" containerName="istio-proxy"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938579    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58596f07-7f21-46b6-b5e4-f8703e4208f8" containerName="istio-proxy"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938584    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="58596f07-7f21-46b6-b5e4-f8703e4208f8" containerName="istio-proxy"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938665    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="58596f07-7f21-46b6-b5e4-f8703e4208f8" containerName="istio-proxy"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938674    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91770344-218b-405e-acc0-f371882355f2" containerName="main"
Sep 17 18:58:45.938730 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.938682    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1999d77-165e-40d3-b3e7-8d36b0f77045" containerName="istio-proxy"
Sep 17 18:58:45.940634 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.940618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.949806 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.949773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:58:45.950872 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.950840    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:58:45.951007 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.950918    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-42x8z\""
Sep 17 18:58:45.951779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.951763    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t76vb\""
Sep 17 18:58:45.951862 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.951811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:58:45.965663 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.965639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:58:45.971264 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmvg\" (UniqueName: \"kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.971364 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.971428 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.971428 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971420    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.971536 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:45.971536 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:45.971499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.071912 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.071871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.071918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.071944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmvg\" (UniqueName: \"kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072022 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.071988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072146 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.072023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072230 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.072211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072288 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.072270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072339 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.072329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.072498 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.072475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.074330 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.074312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.074484 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.074468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.084105 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.084083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmvg\" (UniqueName: \"kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.251122 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.251047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:46.535349 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.535321    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:58:46.536978 ip-10-0-136-146 kubenswrapper[2561]: W0917 18:58:46.536951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce63368a_41cd_4c12_a4be_72ae518d0890.slice/crio-567ec76b86de9238f7690d7aee8e8ad6e3a2441eb7ec02965d34b5a8acf262d6 WatchSource:0}: Error finding container 567ec76b86de9238f7690d7aee8e8ad6e3a2441eb7ec02965d34b5a8acf262d6: Status 404 returned error can't find the container with id 567ec76b86de9238f7690d7aee8e8ad6e3a2441eb7ec02965d34b5a8acf262d6
Sep 17 18:58:46.539053 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:46.539033    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:58:47.095762 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:47.095725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerStarted","Data":"567ec76b86de9238f7690d7aee8e8ad6e3a2441eb7ec02965d34b5a8acf262d6"}
Sep 17 18:58:49.106801 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:49.106764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerStarted","Data":"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"}
Sep 17 18:58:49.107223 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:49.106905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:50.112873 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:50.112840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerStarted","Data":"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"}
Sep 17 18:58:50.143688 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:50.143639    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" podStartSLOduration=3.475236589 podStartE2EDuration="5.143623671s" totalImagesPullingTime="1.668387082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:50.141178872 +0000 UTC m=+1147.282605085" watchObservedRunningTime="2026-09-17 18:58:50.143623671 +0000 UTC m=+1147.285049883"
Sep 17 18:58:50.251576 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:50.251549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:56.251857 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:56.251776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:56.264138 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:56.264114    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:57.155952 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:57.155859    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:58:58.161026 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:58:58.160994    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:59:21.308136 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.308106    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:59:21.317584 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:59:21.317547    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:59:21.317584 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.317584 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.317584 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65" cmd=["/bin/sleep","15"]
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.317605    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" message="" err=<
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:59:21.317617    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="main" containerID="cri-o://c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"
Sep 17 18:59:21.317797 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.317664    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="main" containerID="cri-o://c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65" gracePeriod=60
Sep 17 18:59:21.430971 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.430938    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="llm-d-routing-sidecar" containerID="cri-o://329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11" gracePeriod=60
Sep 17 18:59:21.680587 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.680563    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:59:21.770279 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770436 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770400    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcmvg\" (UniqueName: \"kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770604 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770445    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770604 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache\") pod \"ce63368a-41cd-4c12-a4be-72ae518d0890\" (UID: \"ce63368a-41cd-4c12-a4be-72ae518d0890\") "
Sep 17 18:59:21.770771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770653    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home" (OuterVolumeSpecName: "home") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:21.770771 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:21.770873 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770833    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:21.770873 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770846    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.770873 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.770865    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.772580 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.772562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm" (OuterVolumeSpecName: "dshm") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:21.773068 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.773045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg" (OuterVolumeSpecName: "kube-api-access-xcmvg") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "kube-api-access-xcmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:21.773068 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.773060    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce63368a-41cd-4c12-a4be-72ae518d0890" (UID: "ce63368a-41cd-4c12-a4be-72ae518d0890"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:21.871588 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.871515    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce63368a-41cd-4c12-a4be-72ae518d0890-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.871588 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.871540    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmvg\" (UniqueName: \"kubernetes.io/projected/ce63368a-41cd-4c12-a4be-72ae518d0890-kube-api-access-xcmvg\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.871588 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.871549    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.871588 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:21.871557    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce63368a-41cd-4c12-a4be-72ae518d0890-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.257320 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257225    2561 generic.go:356] "Generic (PLEG): container finished" podID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65" exitCode=0
Sep 17 18:59:22.257320 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257256    2561 generic.go:356] "Generic (PLEG): container finished" podID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerID="329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11" exitCode=0
Sep 17 18:59:22.257320 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257278    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"
Sep 17 18:59:22.257565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerDied","Data":"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"}
Sep 17 18:59:22.257565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerDied","Data":"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"}
Sep 17 18:59:22.257565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m" event={"ID":"ce63368a-41cd-4c12-a4be-72ae518d0890","Type":"ContainerDied","Data":"567ec76b86de9238f7690d7aee8e8ad6e3a2441eb7ec02965d34b5a8acf262d6"}
Sep 17 18:59:22.257565 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.257418    2561 scope.go:117] "RemoveContainer" containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"
Sep 17 18:59:22.269923 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.269887    2561 scope.go:117] "RemoveContainer" containerID="329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"
Sep 17 18:59:22.280200 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.280175    2561 scope.go:117] "RemoveContainer" containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"
Sep 17 18:59:22.280859 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:59:22.280832    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65\": container with ID starting with c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65 not found: ID does not exist" containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"
Sep 17 18:59:22.280990 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.280868    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"} err="failed to get container status \"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65\": rpc error: code = NotFound desc = could not find container \"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65\": container with ID starting with c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65 not found: ID does not exist"
Sep 17 18:59:22.281065 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281006    2561 scope.go:117] "RemoveContainer" containerID="329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"
Sep 17 18:59:22.281310 ip-10-0-136-146 kubenswrapper[2561]: E0917 18:59:22.281291    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11\": container with ID starting with 329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11 not found: ID does not exist" containerID="329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"
Sep 17 18:59:22.281394 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281318    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"} err="failed to get container status \"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11\": rpc error: code = NotFound desc = could not find container \"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11\": container with ID starting with 329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11 not found: ID does not exist"
Sep 17 18:59:22.281394 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281339    2561 scope.go:117] "RemoveContainer" containerID="c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"
Sep 17 18:59:22.281649 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281630    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65"} err="failed to get container status \"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65\": rpc error: code = NotFound desc = could not find container \"c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65\": container with ID starting with c662165d8e11bbfb055f87428c95c548361586d447eb98557cb9bd74caecce65 not found: ID does not exist"
Sep 17 18:59:22.281715 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281651    2561 scope.go:117] "RemoveContainer" containerID="329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"
Sep 17 18:59:22.281967 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.281943    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11"} err="failed to get container status \"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11\": rpc error: code = NotFound desc = could not find container \"329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11\": container with ID starting with 329626e7c40968e38e430eb6d1b1e366ce20ae2d75bb97a2139fb76166a5af11 not found: ID does not exist"
Sep 17 18:59:22.285299 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.285276    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:59:22.289154 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:22.289136    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-7nj7m"]
Sep 17 18:59:23.086479 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:23.086447    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" path="/var/lib/kubelet/pods/ce63368a-41cd-4c12-a4be-72ae518d0890/volumes"
Sep 17 18:59:45.019779 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:45.019752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 18:59:45.050457 ip-10-0-136-146 kubenswrapper[2561]: I0917 18:59:45.050437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:00:38.639112 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639032    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="main"
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639337    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="main"
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639354    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="llm-d-routing-sidecar"
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="llm-d-routing-sidecar"
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639417    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="llm-d-routing-sidecar"
Sep 17 19:00:38.639583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.639424    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce63368a-41cd-4c12-a4be-72ae518d0890" containerName="main"
Sep 17 19:00:38.642369 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.642349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.647499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.647477    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 19:00:38.648499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.648479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 19:00:38.648971 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.648485    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-hrxhq\""
Sep 17 19:00:38.648971 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.648485    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 19:00:38.648971 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.648518    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 19:00:38.648971 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.648533    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:00:38.658638 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.658992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.659093 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.658995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzng\" (UniqueName: \"kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.660047 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.660025    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:00:38.760288 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760422 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760422 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760422 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzng\" (UniqueName: \"kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760722 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760722 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.760722 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.760692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.761089 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.761057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.761192 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.761128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.761300 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.761274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.763000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.762981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.763101 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.763079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.768756 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.768726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzng\" (UniqueName: \"kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-drfmf\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:38.952703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:38.952626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:39.229617 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:39.229583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:00:39.232433 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:00:39.232401    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d519c49_1df9_43dc_ab39_58058d869ad3.slice/crio-dff45ee835b8e5d0080334cce1a0b325f26b20d6f2870c042a51376bb977649e WatchSource:0}: Error finding container dff45ee835b8e5d0080334cce1a0b325f26b20d6f2870c042a51376bb977649e: Status 404 returned error can't find the container with id dff45ee835b8e5d0080334cce1a0b325f26b20d6f2870c042a51376bb977649e
Sep 17 19:00:39.591464 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:39.591425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerStarted","Data":"dff45ee835b8e5d0080334cce1a0b325f26b20d6f2870c042a51376bb977649e"}
Sep 17 19:00:43.615551 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:43.615516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerStarted","Data":"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"}
Sep 17 19:00:46.631279 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:46.631244    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerID="16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47" exitCode=0
Sep 17 19:00:46.631690 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:46.631314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerDied","Data":"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"}
Sep 17 19:00:47.637983 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:47.637946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerStarted","Data":"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"}
Sep 17 19:00:47.666853 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:47.666788    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" podStartSLOduration=2.872629798 podStartE2EDuration="9.666772515s" totalImagesPullingTime="4.169519132s" totalInitContainerRuntime="2.624623585s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:47.664324669 +0000 UTC m=+1264.805750878" watchObservedRunningTime="2026-09-17 19:00:47.666772515 +0000 UTC m=+1264.808198726"
Sep 17 19:00:47.953661 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:47.953575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:48.953387 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:48.953349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:48.966277 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:48.966249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:00:49.658888 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:00:49.658854    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:01:03.226413 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.226379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:01:03.229905 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.229877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.232638 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.232601    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 19:01:03.232802 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.232764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 19:01:03.232954 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.232938    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8xphn\""
Sep 17 19:01:03.233012 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.232962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 19:01:03.233057 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.233028    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:01:03.233433 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.233416    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 19:01:03.247122 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.247100    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:01:03.276963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.276934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277105 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.276992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277105 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46cg\" (UniqueName: \"kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277311 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.277311 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.277239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377801 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46cg\" (UniqueName: \"kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.377981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.377940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378217 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378217 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378353 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378563 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.378793 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.378677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.380499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.380474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.380585 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.380518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.388534 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.388514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46cg\" (UniqueName: \"kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg\") pod \"auth-invalid-token-test-kserve-56c445d97d-46w5q\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.541420 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.541330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:03.826260 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:03.826179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:01:03.830976 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:01:03.830926    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a3c2e3_e040_474f_b387_c9552ca5838f.slice/crio-53387a225a982408911e9115ec1c9fd2f2200be0286c256f0c64ede9d67b83bc WatchSource:0}: Error finding container 53387a225a982408911e9115ec1c9fd2f2200be0286c256f0c64ede9d67b83bc: Status 404 returned error can't find the container with id 53387a225a982408911e9115ec1c9fd2f2200be0286c256f0c64ede9d67b83bc
Sep 17 19:01:04.716356 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:04.716316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerStarted","Data":"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"}
Sep 17 19:01:04.716808 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:04.716388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerStarted","Data":"53387a225a982408911e9115ec1c9fd2f2200be0286c256f0c64ede9d67b83bc"}
Sep 17 19:01:06.727353 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:06.727319    2561 generic.go:356] "Generic (PLEG): container finished" podID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerID="b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f" exitCode=0
Sep 17 19:01:06.727756 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:06.727363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerDied","Data":"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"}
Sep 17 19:01:36.899820 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:36.899769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerStarted","Data":"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"}
Sep 17 19:01:36.934386 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:36.934326    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podStartSLOduration=1.886188859 podStartE2EDuration="33.934302756s" totalImagesPullingTime="29.639489191s" totalInitContainerRuntime="2.408624706s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:36.932670605 +0000 UTC m=+1314.074096819" watchObservedRunningTime="2026-09-17 19:01:36.934302756 +0000 UTC m=+1314.075728970"
Sep 17 19:01:37.542411 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:37.542378    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:43.542275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:43.542240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:01:43.543796 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:43.543769    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:01:53.542518 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:01:53.542470    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:03.483566 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.483516    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:02:03.494504 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:02:03.494467    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:02:03.494504 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.494504 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.494504 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1" cmd=["/bin/sleep","15"]
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.494528    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" message="" err=<
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:02:03.494547    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main" containerID="cri-o://de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"
Sep 17 19:02:03.494774 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.494591    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main" containerID="cri-o://de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1" gracePeriod=60
Sep 17 19:02:03.542666 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.542631    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:03.647948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.647887    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 19:02:03.856123 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.856088    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:02:03.960887 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.960853    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.960887 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.960889    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.960924    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.960989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961064    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kzng\" (UniqueName: \"kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir\") pod \"7d519c49-1df9-43dc-ab39-58058d869ad3\" (UID: \"7d519c49-1df9-43dc-ab39-58058d869ad3\") "
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home" (OuterVolumeSpecName: "home") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:03.961179 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961129    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:03.961606 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961369    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:03.961606 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961385    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:03.961606 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961559    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:03.961606 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.961579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:03.963718 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.963693    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:03.964077 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.964055    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm" (OuterVolumeSpecName: "dshm") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:03.964077 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.964069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng" (OuterVolumeSpecName: "kube-api-access-9kzng") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "kube-api-access-9kzng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:03.993408 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:03.993371    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d519c49-1df9-43dc-ab39-58058d869ad3" (UID: "7d519c49-1df9-43dc-ab39-58058d869ad3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.035528 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.035445    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerID="de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1" exitCode=0
Sep 17 19:02:04.035528 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.035502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"
Sep 17 19:02:04.035797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.035500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerDied","Data":"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"}
Sep 17 19:02:04.035797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.035636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf" event={"ID":"7d519c49-1df9-43dc-ab39-58058d869ad3","Type":"ContainerDied","Data":"dff45ee835b8e5d0080334cce1a0b325f26b20d6f2870c042a51376bb977649e"}
Sep 17 19:02:04.035797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.035667    2561 scope.go:117] "RemoveContainer" containerID="de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"
Sep 17 19:02:04.045509 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.045488    2561 scope.go:117] "RemoveContainer" containerID="16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"
Sep 17 19:02:04.055166 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.055145    2561 scope.go:117] "RemoveContainer" containerID="de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"
Sep 17 19:02:04.055522 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:02:04.055486    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1\": container with ID starting with de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1 not found: ID does not exist" containerID="de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"
Sep 17 19:02:04.055582 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.055535    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1"} err="failed to get container status \"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1\": rpc error: code = NotFound desc = could not find container \"de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1\": container with ID starting with de414ef03ed64aa6b01c02b771bc3bf28801fc10bd08f5bc84a4777a90da19e1 not found: ID does not exist"
Sep 17 19:02:04.055582 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.055560    2561 scope.go:117] "RemoveContainer" containerID="16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"
Sep 17 19:02:04.055878 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:02:04.055859    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47\": container with ID starting with 16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47 not found: ID does not exist" containerID="16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"
Sep 17 19:02:04.056029 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.055884    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47"} err="failed to get container status \"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47\": rpc error: code = NotFound desc = could not find container \"16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47\": container with ID starting with 16e07a37836829469a962f8ef42a17654b88780acd36f1a43e14141f6ed9cd47 not found: ID does not exist"
Sep 17 19:02:04.062379 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062357    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062379 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062381    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d519c49-1df9-43dc-ab39-58058d869ad3-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062505 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062390    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062505 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062399    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d519c49-1df9-43dc-ab39-58058d869ad3-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062505 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062407    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzng\" (UniqueName: \"kubernetes.io/projected/7d519c49-1df9-43dc-ab39-58058d869ad3-kube-api-access-9kzng\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062505 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062436    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d519c49-1df9-43dc-ab39-58058d869ad3-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.062744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.062725    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:02:04.066803 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:04.066780    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-drfmf"]
Sep 17 19:02:05.086328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:05.086296    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" path="/var/lib/kubelet/pods/7d519c49-1df9-43dc-ab39-58058d869ad3/volumes"
Sep 17 19:02:13.542402 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:13.542361    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:23.542191 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:23.542150    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:33.542242 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:33.542195    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:43.542648 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:43.542605    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:02:53.542108 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:02:53.542058    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:03:03.542410 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:03.542367    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:03:04.605405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605368    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:04.605850 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605828    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main"
Sep 17 19:03:04.605850 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605843    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main"
Sep 17 19:03:04.605953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="storage-initializer"
Sep 17 19:03:04.605953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="storage-initializer"
Sep 17 19:03:04.605953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.605944    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d519c49-1df9-43dc-ab39-58058d869ad3" containerName="main"
Sep 17 19:03:04.607887 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.607869    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.612163 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.612135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 19:03:04.612309 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.612178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-4nf2q\""
Sep 17 19:03:04.612933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.612888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 19:03:04.613062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.612952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 19:03:04.613062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.612915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:03:04.613062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.613024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 19:03:04.625765 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.625742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:04.703872 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.703833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.703872 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.703868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.703990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.704036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.704104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.704133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.704159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.704326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.704187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767ft\" (UniqueName: \"kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.804735 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.804735 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767ft\" (UniqueName: \"kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.804993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.805034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805340 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.805180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805397 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.805354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805462 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.805441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.805528 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.805505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.806120 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.806097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.807440 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.807417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.807824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.807803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.814335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.814312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767ft\" (UniqueName: \"kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:04.918737 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:04.918658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:05.199669 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:05.199584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:05.203124 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:03:05.203092    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dac0cb_ef6f_4f71_94f9_5ce92425a312.slice/crio-0a3ae32b0d49f4d22445d70cce8ec131ec495d44f930b9a83b7494ceb284a8d4 WatchSource:0}: Error finding container 0a3ae32b0d49f4d22445d70cce8ec131ec495d44f930b9a83b7494ceb284a8d4: Status 404 returned error can't find the container with id 0a3ae32b0d49f4d22445d70cce8ec131ec495d44f930b9a83b7494ceb284a8d4
Sep 17 19:03:05.321205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:05.321167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerStarted","Data":"805e7a3a0019d6e56161aff6253baba7fa7fffa79d56c64ce8902d11d69d615e"}
Sep 17 19:03:05.321395 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:05.321231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerStarted","Data":"0a3ae32b0d49f4d22445d70cce8ec131ec495d44f930b9a83b7494ceb284a8d4"}
Sep 17 19:03:08.339810 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:08.339768    2561 generic.go:356] "Generic (PLEG): container finished" podID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerID="805e7a3a0019d6e56161aff6253baba7fa7fffa79d56c64ce8902d11d69d615e" exitCode=0
Sep 17 19:03:08.339810 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:08.339812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerDied","Data":"805e7a3a0019d6e56161aff6253baba7fa7fffa79d56c64ce8902d11d69d615e"}
Sep 17 19:03:09.347102 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:09.347067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerStarted","Data":"bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4"}
Sep 17 19:03:09.919732 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:09.919680    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:13.542463 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:13.542411    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:03:14.919884 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:14.919842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:14.936640 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:14.936610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:14.959191 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:14.959133    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" podStartSLOduration=8.335272409 podStartE2EDuration="10.959118268s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.623845859s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:09.38117858 +0000 UTC m=+1406.522604792" watchObservedRunningTime="2026-09-17 19:03:14.959118268 +0000 UTC m=+1412.100544480"
Sep 17 19:03:15.385267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:15.385234    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:23.542243 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:23.542199    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:03:33.542522 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:33.542421    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:03:42.277011 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.274112    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:42.289421 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:03:42.289374    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:03:42.289421 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.289421 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.289421 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4" cmd=["/bin/sleep","15"]
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.289449    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" message="" err=<
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:03:42.289468    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main" containerID="cri-o://bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4"
Sep 17 19:03:42.289720 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.289537    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main" containerID="cri-o://bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4" gracePeriod=60
Sep 17 19:03:42.375472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.375426    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 19:03:42.509172 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.509142    2561 generic.go:356] "Generic (PLEG): container finished" podID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerID="bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4" exitCode=0
Sep 17 19:03:42.509275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.509193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerDied","Data":"bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4"}
Sep 17 19:03:42.636680 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.636653    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:42.762813 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762779    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.762813 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.762971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-767ft\" (UniqueName: \"kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763122    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache" (OuterVolumeSpecName: "model-cache") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:42.763284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763170    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:42.763284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home" (OuterVolumeSpecName: "home") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763298    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763376    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location\") pod \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\" (UID: \"17dac0cb-ef6f-4f71-94f9-5ce92425a312\") "
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763649    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763667    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763681    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17dac0cb-ef6f-4f71-94f9-5ce92425a312-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.763769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.763694    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.765847 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.765809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm" (OuterVolumeSpecName: "dshm") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:42.765985 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.765887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:42.766345 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.766313    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft" (OuterVolumeSpecName: "kube-api-access-767ft") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "kube-api-access-767ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:42.783773 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.783743    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17dac0cb-ef6f-4f71-94f9-5ce92425a312" (UID: "17dac0cb-ef6f-4f71-94f9-5ce92425a312"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:42.864737 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.864697    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dac0cb-ef6f-4f71-94f9-5ce92425a312-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.864737 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.864736    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.865008 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.864753    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dac0cb-ef6f-4f71-94f9-5ce92425a312-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:42.865008 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:42.864769    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767ft\" (UniqueName: \"kubernetes.io/projected/17dac0cb-ef6f-4f71-94f9-5ce92425a312-kube-api-access-767ft\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.515616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.515589    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"
Sep 17 19:03:43.516195 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.515582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg" event={"ID":"17dac0cb-ef6f-4f71-94f9-5ce92425a312","Type":"ContainerDied","Data":"0a3ae32b0d49f4d22445d70cce8ec131ec495d44f930b9a83b7494ceb284a8d4"}
Sep 17 19:03:43.516195 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.515746    2561 scope.go:117] "RemoveContainer" containerID="bf98bf7d96a7a189ee966ad5c19d3376f3c2fc9affeb78967ac22bfbbed71fe4"
Sep 17 19:03:43.524844 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.524821    2561 scope.go:117] "RemoveContainer" containerID="805e7a3a0019d6e56161aff6253baba7fa7fffa79d56c64ce8902d11d69d615e"
Sep 17 19:03:43.540993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.540968    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:43.550514 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.550489    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-vdmrg"]
Sep 17 19:03:43.553149 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.553130    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:03:43.561637 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:43.561611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:03:45.087117 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:45.087075    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" path="/var/lib/kubelet/pods/17dac0cb-ef6f-4f71-94f9-5ce92425a312/volumes"
Sep 17 19:03:46.255369 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:03:46.255338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:04:01.280549 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:01.280505    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" containerID="cri-o://8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10" gracePeriod=45
Sep 17 19:04:01.554767 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:01.554673    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 19:04:02.034691 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.034656    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:04:02.146400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d46cg\" (UniqueName: \"kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146610    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146930 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.146930 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.146724    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache\") pod \"b3a3c2e3-e040-474f-b387-c9552ca5838f\" (UID: \"b3a3c2e3-e040-474f-b387-c9552ca5838f\") "
Sep 17 19:04:02.147084 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.147056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:02.147269 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.147231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home" (OuterVolumeSpecName: "home") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:02.147269 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.147256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:02.148960 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.148931    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:02.149393 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.149368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm" (OuterVolumeSpecName: "dshm") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:02.149393 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.149384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg" (OuterVolumeSpecName: "kube-api-access-d46cg") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "kube-api-access-d46cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:02.158470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.158446    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:02.174333 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.174296    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3a3c2e3-e040-474f-b387-c9552ca5838f" (UID: "b3a3c2e3-e040-474f-b387-c9552ca5838f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:02.247866 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247836    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.247866 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247865    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3a3c2e3-e040-474f-b387-c9552ca5838f-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247873    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247880    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247889    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46cg\" (UniqueName: \"kubernetes.io/projected/b3a3c2e3-e040-474f-b387-c9552ca5838f-kube-api-access-d46cg\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247924    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247933    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3a3c2e3-e040-474f-b387-c9552ca5838f-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.248037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.247941    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3a3c2e3-e040-474f-b387-c9552ca5838f-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:04:02.605476 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.605445    2561 generic.go:356] "Generic (PLEG): container finished" podID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerID="8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10" exitCode=0
Sep 17 19:04:02.605862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.605498    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"
Sep 17 19:04:02.605862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.605514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerDied","Data":"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"}
Sep 17 19:04:02.605862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.605573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q" event={"ID":"b3a3c2e3-e040-474f-b387-c9552ca5838f","Type":"ContainerDied","Data":"53387a225a982408911e9115ec1c9fd2f2200be0286c256f0c64ede9d67b83bc"}
Sep 17 19:04:02.605862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.605598    2561 scope.go:117] "RemoveContainer" containerID="8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"
Sep 17 19:04:02.618320 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.618295    2561 scope.go:117] "RemoveContainer" containerID="b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"
Sep 17 19:04:02.626270 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.626249    2561 scope.go:117] "RemoveContainer" containerID="8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"
Sep 17 19:04:02.626529 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:04:02.626512    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10\": container with ID starting with 8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10 not found: ID does not exist" containerID="8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"
Sep 17 19:04:02.626576 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.626538    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10"} err="failed to get container status \"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10\": rpc error: code = NotFound desc = could not find container \"8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10\": container with ID starting with 8a5dec33ff6dfae0b4a96a253b72ae647a56435de4d312ad1cdbc5ad298a6a10 not found: ID does not exist"
Sep 17 19:04:02.626576 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.626558    2561 scope.go:117] "RemoveContainer" containerID="b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"
Sep 17 19:04:02.626834 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:04:02.626816    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f\": container with ID starting with b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f not found: ID does not exist" containerID="b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"
Sep 17 19:04:02.626888 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:02.626841    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f"} err="failed to get container status \"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f\": rpc error: code = NotFound desc = could not find container \"b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f\": container with ID starting with b744b0922cfd7c13a0f9733cbca762e4a94aa6fa5a0bfbe0145fb32a4122af7f not found: ID does not exist"
Sep 17 19:04:03.336099 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:03.336059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:04:03.339438 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:03.339407    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56c445d97d-46w5q"]
Sep 17 19:04:05.087086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:05.087059    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" path="/var/lib/kubelet/pods/b3a3c2e3-e040-474f-b387-c9552ca5838f/volumes"
Sep 17 19:04:45.053230 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:45.053199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:04:45.075679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:45.075657    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:04:57.591142 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591439    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="storage-initializer"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="storage-initializer"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591467    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591473    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591485    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="storage-initializer"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="storage-initializer"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591496    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591502    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main"
Sep 17 19:04:57.591550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591551    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a3c2e3-e040-474f-b387-c9552ca5838f" containerName="main"
Sep 17 19:04:57.591855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.591561    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dac0cb-ef6f-4f71-94f9-5ce92425a312" containerName="main"
Sep 17 19:04:57.595864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.595841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.599372 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.599351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 19:04:57.599476 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.599462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-gzg4z\""
Sep 17 19:04:57.600349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.600318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 19:04:57.600349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.600343    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 19:04:57.600511 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.600370    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 19:04:57.600511 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.600380    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:04:57.609487 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.609461    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:04:57.634329 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jnv\" (UniqueName: \"kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.634683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.634671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735450 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735701 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735701 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jnv\" (UniqueName: \"kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.735806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.736005 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.736005 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.735953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.736109 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.736041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.736149 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.736120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.736393 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.736371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.738102 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.738068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.738202 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.738181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.745717 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.745694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jnv\" (UniqueName: \"kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-ctgzf\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:57.906571 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:57.906484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:04:58.190505 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:58.190473    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:04:58.192229 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:04:58.192203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46b800_c9b0_4e59_a7e6_f45cfa03c0c4.slice/crio-6cd12e6d5a48193890ac6e1c8bf411ffa063fa314919c77305e1ea6e22216e87 WatchSource:0}: Error finding container 6cd12e6d5a48193890ac6e1c8bf411ffa063fa314919c77305e1ea6e22216e87: Status 404 returned error can't find the container with id 6cd12e6d5a48193890ac6e1c8bf411ffa063fa314919c77305e1ea6e22216e87
Sep 17 19:04:58.855633 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:58.855586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerStarted","Data":"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"}
Sep 17 19:04:58.856160 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:04:58.855696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerStarted","Data":"6cd12e6d5a48193890ac6e1c8bf411ffa063fa314919c77305e1ea6e22216e87"}
Sep 17 19:05:01.873468 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:01.873438    2561 generic.go:356] "Generic (PLEG): container finished" podID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerID="db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b" exitCode=0
Sep 17 19:05:01.873468 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:01.873482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerDied","Data":"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"}
Sep 17 19:05:02.879596 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:02.879559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerStarted","Data":"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"}
Sep 17 19:05:02.907006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:02.906975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:02.918096 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:02.918043    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" podStartSLOduration=3.143153708 podStartE2EDuration="5.918017721s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.774864013s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:02.915222046 +0000 UTC m=+1520.056648255" watchObservedRunningTime="2026-09-17 19:05:02.918017721 +0000 UTC m=+1520.059443934"
Sep 17 19:05:07.907354 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:07.907319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:07.919811 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:07.919785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:08.918413 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:08.918383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:18.008482 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.008448    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:05:18.011567 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.011543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.016188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.016160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 19:05:18.016339 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.016259    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 19:05:18.016734 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.016712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:05:18.017069 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.017053    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 19:05:18.017142 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.017068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-bk9g9\""
Sep 17 19:05:18.017142 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.017059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 19:05:18.033560 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.033535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:05:18.129099 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.129295 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.129294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8q6l\" (UniqueName: \"kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230669 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8q6l\" (UniqueName: \"kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.230880 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231342 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.230928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231342 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.231136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231342 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.231252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231342 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.231321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231489 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.231476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.231575 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.231555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.233548 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.233518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.233797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.233779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.242628 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.242604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8q6l\" (UniqueName: \"kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l\") pod \"auth-disabled-test-kserve-698977d599-5mnst\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.322652 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.322555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:18.602847 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.602821    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:05:18.605495 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:05:18.605460    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f11a89_8770_4fd3_ae7d_62b7c2a3b279.slice/crio-810cac1f8faf7191d0ea3acca0747d5f53672968beeef8f4bd2f0afe6d2cb3ca WatchSource:0}: Error finding container 810cac1f8faf7191d0ea3acca0747d5f53672968beeef8f4bd2f0afe6d2cb3ca: Status 404 returned error can't find the container with id 810cac1f8faf7191d0ea3acca0747d5f53672968beeef8f4bd2f0afe6d2cb3ca
Sep 17 19:05:18.956458 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.956344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerStarted","Data":"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"}
Sep 17 19:05:18.956631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:18.956465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerStarted","Data":"810cac1f8faf7191d0ea3acca0747d5f53672968beeef8f4bd2f0afe6d2cb3ca"}
Sep 17 19:05:21.971856 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:21.971824    2561 generic.go:356] "Generic (PLEG): container finished" podID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerID="182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031" exitCode=0
Sep 17 19:05:21.972281 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:21.971861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerDied","Data":"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"}
Sep 17 19:05:22.978634 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:22.978595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerStarted","Data":"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"}
Sep 17 19:05:23.005993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:23.005942    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podStartSLOduration=3.563849596 podStartE2EDuration="6.005925626s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.44207603s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:23.004747478 +0000 UTC m=+1540.146173690" watchObservedRunningTime="2026-09-17 19:05:23.005925626 +0000 UTC m=+1540.147351832"
Sep 17 19:05:23.323536 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:23.323447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:28.322819 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:28.322779    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:05:28.324246 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:28.324202    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:05:38.323199 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:38.323151    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:05:48.323229 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:48.323185    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:05:52.296801 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.296556    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:05:52.309852    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419" cmd=["/bin/sleep","15"]
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.309920    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" message="" err=<
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:05:52.309935    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="main" containerID="cri-o://1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"
Sep 17 19:05:52.310111 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.309972    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="main" containerID="cri-o://1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419" gracePeriod=60
Sep 17 19:05:52.669034 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.669011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:52.760797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760764    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7jnv\" (UniqueName: \"kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.760951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761038    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm\") pod \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\" (UID: \"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4\") "
Sep 17 19:05:52.761355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761164    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.761355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.761355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home" (OuterVolumeSpecName: "home") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.761355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761225    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:52.761627 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761465    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.761627 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761483    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.761627 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761493    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.761627 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.761501    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.764126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.764094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv" (OuterVolumeSpecName: "kube-api-access-n7jnv") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "kube-api-access-n7jnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:52.764126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.764120    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm" (OuterVolumeSpecName: "dshm") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.764277 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.764122    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:52.784275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.784236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" (UID: "fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.862609 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.862574    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.862609 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.862604    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.862609 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.862615    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jnv\" (UniqueName: \"kubernetes.io/projected/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-kube-api-access-n7jnv\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:52.862843 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:52.862625    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.119879 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.119800    2561 generic.go:356] "Generic (PLEG): container finished" podID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerID="1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419" exitCode=0
Sep 17 19:05:53.119879 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.119859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerDied","Data":"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"}
Sep 17 19:05:53.119879 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.119861    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"
Sep 17 19:05:53.120154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.119927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf" event={"ID":"fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4","Type":"ContainerDied","Data":"6cd12e6d5a48193890ac6e1c8bf411ffa063fa314919c77305e1ea6e22216e87"}
Sep 17 19:05:53.120154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.119951    2561 scope.go:117] "RemoveContainer" containerID="1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"
Sep 17 19:05:53.129432 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.129404    2561 scope.go:117] "RemoveContainer" containerID="db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"
Sep 17 19:05:53.137718 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.137698    2561 scope.go:117] "RemoveContainer" containerID="1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"
Sep 17 19:05:53.138000 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:05:53.137979    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419\": container with ID starting with 1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419 not found: ID does not exist" containerID="1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"
Sep 17 19:05:53.138075 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.138007    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419"} err="failed to get container status \"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419\": rpc error: code = NotFound desc = could not find container \"1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419\": container with ID starting with 1c6942af11b8f7f9a471ff28d40ceae41efddbee929519598a9bb083277fb419 not found: ID does not exist"
Sep 17 19:05:53.138075 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.138038    2561 scope.go:117] "RemoveContainer" containerID="db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"
Sep 17 19:05:53.138340 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:05:53.138317    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b\": container with ID starting with db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b not found: ID does not exist" containerID="db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"
Sep 17 19:05:53.138407 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.138338    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b"} err="failed to get container status \"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b\": rpc error: code = NotFound desc = could not find container \"db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b\": container with ID starting with db0881b0800dff8c42e9e439756721e4dd0890b124f8c9341b7e0ac1cbd1979b not found: ID does not exist"
Sep 17 19:05:53.142409 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.142386    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:05:53.148380 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:53.148359    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-ctgzf"]
Sep 17 19:05:55.088147 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:55.088114    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" path="/var/lib/kubelet/pods/fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4/volumes"
Sep 17 19:05:58.323470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:05:58.323429    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:08.323541 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:08.323491    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:18.323793 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:18.323742    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:28.323959 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:28.323821    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:38.324071 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:38.324024    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:48.323598 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:48.323546    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:06:58.323392 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:06:58.323338    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:07:08.324167 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:07:08.324119    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:07:18.323439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:07:18.323383    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:07:28.334330 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:07:28.334293    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:07:28.342839 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:07:28.342807    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:07:56.135194 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:07:56.135154    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:08:11.160516 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:11.160470    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" containerID="cri-o://361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce" gracePeriod=45
Sep 17 19:08:11.335789 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:11.335739    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 17 19:08:11.916759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:11.916727    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:08:12.052649 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052555    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052649 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052649 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8q6l\" (UniqueName: \"kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir\") pod \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\" (UID: \"09f11a89-8770-4fd3-ae7d-62b7c2a3b279\") "
Sep 17 19:08:12.052974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.052938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache" (OuterVolumeSpecName: "model-cache") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.053305 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.053047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:12.053305 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.053210    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.053583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.053516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home" (OuterVolumeSpecName: "home") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.055422 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.055385    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l" (OuterVolumeSpecName: "kube-api-access-r8q6l") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "kube-api-access-r8q6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.055552 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.055462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm" (OuterVolumeSpecName: "dshm") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.055638 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.055617    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:12.065664 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.065636    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.074711 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.074683    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09f11a89-8770-4fd3-ae7d-62b7c2a3b279" (UID: "09f11a89-8770-4fd3-ae7d-62b7c2a3b279"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.153950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153888    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.153950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153942    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.153950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153951    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.153950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153959    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.154229 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153969    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.154229 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153978    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8q6l\" (UniqueName: \"kubernetes.io/projected/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-kube-api-access-r8q6l\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.154229 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.153986    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f11a89-8770-4fd3-ae7d-62b7c2a3b279-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.771792 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.771761    2561 generic.go:356] "Generic (PLEG): container finished" podID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerID="361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce" exitCode=0
Sep 17 19:08:12.772314 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.771814    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerDied","Data":"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"}
Sep 17 19:08:12.772314 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.771839    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"
Sep 17 19:08:12.772314 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.771914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst" event={"ID":"09f11a89-8770-4fd3-ae7d-62b7c2a3b279","Type":"ContainerDied","Data":"810cac1f8faf7191d0ea3acca0747d5f53672968beeef8f4bd2f0afe6d2cb3ca"}
Sep 17 19:08:12.772314 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.771922    2561 scope.go:117] "RemoveContainer" containerID="361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"
Sep 17 19:08:12.782354 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.782322    2561 scope.go:117] "RemoveContainer" containerID="182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"
Sep 17 19:08:12.790798 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.790782    2561 scope.go:117] "RemoveContainer" containerID="361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"
Sep 17 19:08:12.791177 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:08:12.791156    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce\": container with ID starting with 361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce not found: ID does not exist" containerID="361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"
Sep 17 19:08:12.791259 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.791185    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce"} err="failed to get container status \"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce\": rpc error: code = NotFound desc = could not find container \"361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce\": container with ID starting with 361a8a35467b1f90b1e8ab4f661afdc9e3d1a3838685771fc6ec78d669f621ce not found: ID does not exist"
Sep 17 19:08:12.791259 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.791203    2561 scope.go:117] "RemoveContainer" containerID="182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"
Sep 17 19:08:12.791466 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:08:12.791450    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031\": container with ID starting with 182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031 not found: ID does not exist" containerID="182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"
Sep 17 19:08:12.791506 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.791475    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031"} err="failed to get container status \"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031\": rpc error: code = NotFound desc = could not find container \"182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031\": container with ID starting with 182c1e3f5fe2178170046ce5cb00537cc4ce96b6954c75b198c2e4f94011a031 not found: ID does not exist"
Sep 17 19:08:12.797732 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.797709    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:08:12.802043 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:12.802020    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-698977d599-5mnst"]
Sep 17 19:08:13.086864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:13.086836    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" path="/var/lib/kubelet/pods/09f11a89-8770-4fd3-ae7d-62b7c2a3b279/volumes"
Sep 17 19:08:45.382602 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382568    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382912    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="storage-initializer"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382925    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="storage-initializer"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382942    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382947    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="main"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382959    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="main"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382965    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="storage-initializer"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.382970    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="storage-initializer"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.383028    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46b800-c9b0-4e59-a7e6-f45cfa03c0c4" containerName="main"
Sep 17 19:08:45.383041 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.383039    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f11a89-8770-4fd3-ae7d-62b7c2a3b279" containerName="main"
Sep 17 19:08:45.385794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.385773    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.390212 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.390180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:08:45.390353 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.390295    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 19:08:45.391172 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.391155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-9l45p\""
Sep 17 19:08:45.391172 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.391167    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 19:08:45.402735 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.402712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:08:45.464989 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.464881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.464989 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.464947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49l7\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.465204 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.465177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566106 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49l7\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566106 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566676 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566733 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566733 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566733 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566934 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.566934 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.566843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.567103 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.567073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.569068 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.569038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.569278 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.569260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.576623 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.576596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49l7\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.578649 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.578629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-9gsg6\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.696163 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.696118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:45.984153 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.984102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:08:45.987967 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:08:45.987930    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5776d71c_6095_4970_af5b_0f145a7dba51.slice/crio-ef10bf560598e4e5cfe2c634fef20f3f07a39f1e6c1f877dd0d167a44da5abf4 WatchSource:0}: Error finding container ef10bf560598e4e5cfe2c634fef20f3f07a39f1e6c1f877dd0d167a44da5abf4: Status 404 returned error can't find the container with id ef10bf560598e4e5cfe2c634fef20f3f07a39f1e6c1f877dd0d167a44da5abf4
Sep 17 19:08:45.990137 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.990094    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:08:45.990234 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.990180    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:08:45.990234 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:45.990220    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:08:46.930452 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:46.930410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerStarted","Data":"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"}
Sep 17 19:08:46.930862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:46.930494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerStarted","Data":"ef10bf560598e4e5cfe2c634fef20f3f07a39f1e6c1f877dd0d167a44da5abf4"}
Sep 17 19:08:46.953124 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:46.953065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podStartSLOduration=1.953045728 podStartE2EDuration="1.953045728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:46.951010998 +0000 UTC m=+1744.092437211" watchObservedRunningTime="2026-09-17 19:08:46.953045728 +0000 UTC m=+1744.094471940"
Sep 17 19:08:47.697076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:47.697010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:47.702823 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:47.702780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:47.935244 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:47.935210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:08:47.936474 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:08:47.936457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:09:15.283038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.283001    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:15.286097 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.286070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.289429 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.289402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-xz6pb\""
Sep 17 19:09:15.289603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.289448    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 19:09:15.304245 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.304214    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:15.346335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh82\" (UniqueName: \"kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.346520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.346520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.346520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.346632 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.346632 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.346567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.447949 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.447880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448132 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.447965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448132 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh82\" (UniqueName: \"kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448132 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448132 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448132 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448564 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448564 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.448827 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.448798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.451009 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.450981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.451205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.451181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.460752 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.460589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh82\" (UniqueName: \"kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.599063 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.599032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:15.884852 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:15.884763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:15.887488 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:09:15.887448    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23910ff8_21e1_4cf7_942e_5510f88e2ec9.slice/crio-a30956ea9586f3f64224d6f398eff28d0e5d0d9cde560a6e5e6dab0e4d27a60c WatchSource:0}: Error finding container a30956ea9586f3f64224d6f398eff28d0e5d0d9cde560a6e5e6dab0e4d27a60c: Status 404 returned error can't find the container with id a30956ea9586f3f64224d6f398eff28d0e5d0d9cde560a6e5e6dab0e4d27a60c
Sep 17 19:09:16.074353 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:16.074307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" event={"ID":"23910ff8-21e1-4cf7-942e-5510f88e2ec9","Type":"ContainerStarted","Data":"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"}
Sep 17 19:09:16.074553 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:16.074369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" event={"ID":"23910ff8-21e1-4cf7-942e-5510f88e2ec9","Type":"ContainerStarted","Data":"a30956ea9586f3f64224d6f398eff28d0e5d0d9cde560a6e5e6dab0e4d27a60c"}
Sep 17 19:09:16.126605 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:16.126544    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" podStartSLOduration=1.126523453 podStartE2EDuration="1.126523453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:16.123355534 +0000 UTC m=+1773.264781746" watchObservedRunningTime="2026-09-17 19:09:16.126523453 +0000 UTC m=+1773.267949664"
Sep 17 19:09:16.599400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:16.599353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:25.599215 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:25.599177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:25.612014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:25.611979    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:26.120431 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.120396    2561 generic.go:356] "Generic (PLEG): container finished" podID="5776d71c-6095-4970-af5b-0f145a7dba51" containerID="fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd" exitCode=0
Sep 17 19:09:26.120623 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.120471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerDied","Data":"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"}
Sep 17 19:09:26.121036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.121000    2561 scope.go:117] "RemoveContainer" containerID="fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"
Sep 17 19:09:26.122028 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.121990    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:09:26.122105 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.122069    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:09:26.122144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.122101    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:09:26.136382 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:26.136358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:27.129380 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:27.129328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerStarted","Data":"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"}
Sep 17 19:09:27.697243 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:27.697203    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:09:27.697457 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:27.697427    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 17 19:09:28.697511 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:28.697462    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 17 19:09:29.696759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:29.696714    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 17 19:09:30.697223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:30.697175    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 17 19:09:31.697171 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:31.697124    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 17 19:09:32.700868 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:32.700835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:09:32.701284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:32.701138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:09:32.701871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:32.701849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:09:45.082082 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:45.082054    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:09:45.103116 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:45.103091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:09:55.407694 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.405071    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:55.416066 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:09:55.416032    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:09:55.416066 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:09:55.416066 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:09:55.416066 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5" cmd=["/bin/sleep","15"]
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.416082    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" message="" err=<
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:09:55.416094    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerName="main" containerID="cri-o://1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"
Sep 17 19:09:55.416324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.416137    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerName="main" containerID="cri-o://1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5" gracePeriod=60
Sep 17 19:09:55.772755 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.772729    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:55.828149 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828149 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkh82\" (UniqueName: \"kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828410 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828175    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828410 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828363    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828522 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:55.828522 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828640 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home\") pod \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\" (UID: \"23910ff8-21e1-4cf7-942e-5510f88e2ec9\") "
Sep 17 19:09:55.828696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828637    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:55.828928 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828867    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.828928 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home" (OuterVolumeSpecName: "home") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:55.828928 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.828890    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.830953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.830915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82" (OuterVolumeSpecName: "kube-api-access-mkh82") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "kube-api-access-mkh82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:55.831055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.830993    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm" (OuterVolumeSpecName: "dshm") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:55.831055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.830997    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23910ff8-21e1-4cf7-942e-5510f88e2ec9" (UID: "23910ff8-21e1-4cf7-942e-5510f88e2ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:55.929857 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.929817    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23910ff8-21e1-4cf7-942e-5510f88e2ec9-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.929857 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.929859    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkh82\" (UniqueName: \"kubernetes.io/projected/23910ff8-21e1-4cf7-942e-5510f88e2ec9-kube-api-access-mkh82\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.930117 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.929873    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:55.930117 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:55.929889    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23910ff8-21e1-4cf7-942e-5510f88e2ec9-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:09:56.272961 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.272861    2561 generic.go:356] "Generic (PLEG): container finished" podID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerID="1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5" exitCode=0
Sep 17 19:09:56.272961 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.272925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"
Sep 17 19:09:56.272961 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.272945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" event={"ID":"23910ff8-21e1-4cf7-942e-5510f88e2ec9","Type":"ContainerDied","Data":"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"}
Sep 17 19:09:56.273201 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.273003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g" event={"ID":"23910ff8-21e1-4cf7-942e-5510f88e2ec9","Type":"ContainerDied","Data":"a30956ea9586f3f64224d6f398eff28d0e5d0d9cde560a6e5e6dab0e4d27a60c"}
Sep 17 19:09:56.273201 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.273027    2561 scope.go:117] "RemoveContainer" containerID="1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"
Sep 17 19:09:56.283641 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.283621    2561 scope.go:117] "RemoveContainer" containerID="1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"
Sep 17 19:09:56.284049 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:09:56.284026    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5\": container with ID starting with 1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5 not found: ID does not exist" containerID="1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"
Sep 17 19:09:56.284106 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.284061    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5"} err="failed to get container status \"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5\": rpc error: code = NotFound desc = could not find container \"1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5\": container with ID starting with 1a3ecbb8a7919988263031d6bcc24de7272422dd83cc6a356e52a98fd4616fc5 not found: ID does not exist"
Sep 17 19:09:56.297177 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.297155    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:56.301747 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:56.301729    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972c54g"]
Sep 17 19:09:57.086276 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:09:57.086244    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" path="/var/lib/kubelet/pods/23910ff8-21e1-4cf7-942e-5510f88e2ec9/volumes"
Sep 17 19:10:01.769328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:01.769291    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:10:01.769757 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:01.769529    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" containerID="cri-o://0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015" gracePeriod=30
Sep 17 19:10:02.703939 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:02.703866    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:10:07.020134 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.020105    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:10:07.128857 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.128815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.128857 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.128857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.128932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.128955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.128984    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d49l7\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129014    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129048    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129089    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129131 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129117    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo\") pod \"5776d71c-6095-4970-af5b-0f145a7dba51\" (UID: \"5776d71c-6095-4970-af5b-0f145a7dba51\") "
Sep 17 19:10:07.129493 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data" (OuterVolumeSpecName: "istio-data") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:07.129493 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:07.129493 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:07.129647 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:07.129718 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.129694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:07.131688 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.131658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token" (OuterVolumeSpecName: "istio-token") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:07.131850 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.131754    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:10:07.131850 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.131773    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:07.132111 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.132081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7" (OuterVolumeSpecName: "kube-api-access-d49l7") pod "5776d71c-6095-4970-af5b-0f145a7dba51" (UID: "5776d71c-6095-4970-af5b-0f145a7dba51"). InnerVolumeSpecName "kube-api-access-d49l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:07.230794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230751    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-istio-token\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.230794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230789    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5776d71c-6095-4970-af5b-0f145a7dba51-istiod-ca-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.230794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230799    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-data\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230808    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-credential-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230818    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49l7\" (UniqueName: \"kubernetes.io/projected/5776d71c-6095-4970-af5b-0f145a7dba51-kube-api-access-d49l7\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230828    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-istio-envoy\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230836    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230844    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5776d71c-6095-4970-af5b-0f145a7dba51-workload-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.231091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.230852    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5776d71c-6095-4970-af5b-0f145a7dba51-istio-podinfo\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:10:07.325742 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.325708    2561 generic.go:356] "Generic (PLEG): container finished" podID="5776d71c-6095-4970-af5b-0f145a7dba51" containerID="0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015" exitCode=0
Sep 17 19:10:07.325980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.325752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"
Sep 17 19:10:07.325980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.325793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerDied","Data":"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"}
Sep 17 19:10:07.325980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.325870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6" event={"ID":"5776d71c-6095-4970-af5b-0f145a7dba51","Type":"ContainerDied","Data":"ef10bf560598e4e5cfe2c634fef20f3f07a39f1e6c1f877dd0d167a44da5abf4"}
Sep 17 19:10:07.325980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.325917    2561 scope.go:117] "RemoveContainer" containerID="0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"
Sep 17 19:10:07.335998 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.335975    2561 scope.go:117] "RemoveContainer" containerID="fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"
Sep 17 19:10:07.344919 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.344875    2561 scope.go:117] "RemoveContainer" containerID="0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"
Sep 17 19:10:07.345337 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:10:07.345313    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015\": container with ID starting with 0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015 not found: ID does not exist" containerID="0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"
Sep 17 19:10:07.345382 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.345348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015"} err="failed to get container status \"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015\": rpc error: code = NotFound desc = could not find container \"0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015\": container with ID starting with 0e75ad61071fc37a1ccb968d820b04fab1568b038795b74dcdcc1ab1a4167015 not found: ID does not exist"
Sep 17 19:10:07.345382 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.345367    2561 scope.go:117] "RemoveContainer" containerID="fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"
Sep 17 19:10:07.345638 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:10:07.345619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd\": container with ID starting with fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd not found: ID does not exist" containerID="fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"
Sep 17 19:10:07.345702 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.345644    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd"} err="failed to get container status \"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd\": rpc error: code = NotFound desc = could not find container \"fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd\": container with ID starting with fd6e8d85b7c3bdddc7c07f165ed971fcece29634310c6986c4df60de6f2f62dd not found: ID does not exist"
Sep 17 19:10:07.360181 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.360152    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:10:07.368049 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:07.368022    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-9gsg6"]
Sep 17 19:10:09.086592 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:09.086558    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" path="/var/lib/kubelet/pods/5776d71c-6095-4970-af5b-0f145a7dba51/volumes"
Sep 17 19:10:27.666328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666283    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:10:27.666817 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666778    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerName="main"
Sep 17 19:10:27.666817 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666803    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerName="main"
Sep 17 19:10:27.666817 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666817    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.667014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666826    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.667014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666971    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="23910ff8-21e1-4cf7-942e-5510f88e2ec9" containerName="main"
Sep 17 19:10:27.667014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666986    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.667014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.666999    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.667214 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.667078    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.667214 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.667089    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5776d71c-6095-4970-af5b-0f145a7dba51" containerName="istio-proxy"
Sep 17 19:10:27.671936 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.671910    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.674837 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.674809    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 19:10:27.674991 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.674956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:10:27.675612 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.675594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 19:10:27.675806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.675780    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 19:10:27.675965 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.675917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-s8xd8\""
Sep 17 19:10:27.675965 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.675926    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 19:10:27.687419 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.687390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:10:27.708050 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708357 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708357 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvv45\" (UniqueName: \"kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708357 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.708484 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.708357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvv45\" (UniqueName: \"kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.809992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.809919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.810391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.810287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.810391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.810361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.810590 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.810556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.810653 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.810626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.810881 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.810858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.812282 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.812262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.812537 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.812518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.818761 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.818727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvv45\" (UniqueName: \"kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:27.885001 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.884961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:10:27.890174 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.890146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:27.893350 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.893324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-ckgwd\""
Sep 17 19:10:27.899286 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.899255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:10:27.911305 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.911274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:27.911470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.911422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpcg\" (UniqueName: \"kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:27.984040 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:27.983941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:28.012840 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.012792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpcg\" (UniqueName: \"kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:28.013029 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.012885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:28.015753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.015722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:28.022806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.022772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpcg\" (UniqueName: \"kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:28.202737 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.202695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:28.271223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.271183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:10:28.272387 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:10:28.272355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30a9ce8_7798_4ae1_8012_1d820e55234b.slice/crio-55bd53565581ef56853921c6551b45b3adf8820550d2b1b9fed1a09e5e2c5e56 WatchSource:0}: Error finding container 55bd53565581ef56853921c6551b45b3adf8820550d2b1b9fed1a09e5e2c5e56: Status 404 returned error can't find the container with id 55bd53565581ef56853921c6551b45b3adf8820550d2b1b9fed1a09e5e2c5e56
Sep 17 19:10:28.425993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.425685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerStarted","Data":"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"}
Sep 17 19:10:28.425993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.425754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerStarted","Data":"55bd53565581ef56853921c6551b45b3adf8820550d2b1b9fed1a09e5e2c5e56"}
Sep 17 19:10:28.485007 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:28.484741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:10:28.488127 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:10:28.488090    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144342a2_d87f_433f_bf3e_5a0cf3ca150a.slice/crio-d4ee41ba330809160efe064000360d85518c67465c756889c0834ee1972bd91a WatchSource:0}: Error finding container d4ee41ba330809160efe064000360d85518c67465c756889c0834ee1972bd91a: Status 404 returned error can't find the container with id d4ee41ba330809160efe064000360d85518c67465c756889c0834ee1972bd91a
Sep 17 19:10:29.433323 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:29.433282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" event={"ID":"144342a2-d87f-433f-bf3e-5a0cf3ca150a","Type":"ContainerStarted","Data":"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"}
Sep 17 19:10:29.433818 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:29.433354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" event={"ID":"144342a2-d87f-433f-bf3e-5a0cf3ca150a","Type":"ContainerStarted","Data":"d4ee41ba330809160efe064000360d85518c67465c756889c0834ee1972bd91a"}
Sep 17 19:10:29.454080 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:29.452914    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podStartSLOduration=2.452868099 podStartE2EDuration="2.452868099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:29.45151603 +0000 UTC m=+1846.592942245" watchObservedRunningTime="2026-09-17 19:10:29.452868099 +0000 UTC m=+1846.594294311"
Sep 17 19:10:30.438712 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:30.438672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:10:31.447425 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:31.447391    2561 generic.go:356] "Generic (PLEG): container finished" podID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerID="99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398" exitCode=0
Sep 17 19:10:31.448011 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:31.447472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerDied","Data":"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"}
Sep 17 19:10:32.455157 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:32.455115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerStarted","Data":"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"}
Sep 17 19:10:32.477823 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:32.477745    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podStartSLOduration=2.741758291 podStartE2EDuration="5.47771793s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.735959639s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:32.476025226 +0000 UTC m=+1849.617451439" watchObservedRunningTime="2026-09-17 19:10:32.47771793 +0000 UTC m=+1849.619144143"
Sep 17 19:10:32.985004 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:32.984951    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:37.984942 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:37.984875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:10:37.986908 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:37.986852    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:10:47.985373 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:47.985332    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:10:57.985078 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:10:57.985030    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:01.449949 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:01.449888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:11:07.984671 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:07.984622    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:17.984790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:17.984749    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:27.985274 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:27.985227    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:37.984610 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:37.984563    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:47.985030 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:47.984985    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:11:57.984871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:11:57.984825    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:12:07.985108 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:07.985061    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:12:17.985269 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:17.985219    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:12:27.985354 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:27.985257    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:12:37.994927 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:37.994886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:12:38.003089 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:38.003063    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:12:48.744722 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:48.744688    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:12:48.752560 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:48.752534    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:12:51.449044 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:12:51.449005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:01.449197 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:01.449159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:03.765970 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:03.765930    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" containerID="cri-o://3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6" gracePeriod=45
Sep 17 19:13:03.776357 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:03.776306    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" containerID="cri-o://ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee" gracePeriod=45
Sep 17 19:13:03.996112 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:03.996069    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 17 19:13:04.447280 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.447257    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:13:04.470465 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvv45\" (UniqueName: \"kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470582    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470823 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470823 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.470823 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470716    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm\") pod \"f30a9ce8-7798-4ae1-8012-1d820e55234b\" (UID: \"f30a9ce8-7798-4ae1-8012-1d820e55234b\") "
Sep 17 19:13:04.471007 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470944    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:04.471065 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.470995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:04.471496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.471280    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30a9ce8-7798-4ae1-8012-1d820e55234b-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.471496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.471304    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.472302 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.472267    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home" (OuterVolumeSpecName: "home") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:04.473964 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.473921    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:04.474168 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.474144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm" (OuterVolumeSpecName: "dshm") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:04.474409 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.474388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45" (OuterVolumeSpecName: "kube-api-access-gvv45") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "kube-api-access-gvv45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:04.491359 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.491326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:04.501134 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.501103    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f30a9ce8-7798-4ae1-8012-1d820e55234b" (UID: "f30a9ce8-7798-4ae1-8012-1d820e55234b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:04.572000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.571930    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvv45\" (UniqueName: \"kubernetes.io/projected/f30a9ce8-7798-4ae1-8012-1d820e55234b-kube-api-access-gvv45\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.572000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.571952    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.572000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.571980    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.572000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.571990    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30a9ce8-7798-4ae1-8012-1d820e55234b-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.572000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.571999    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:04.572239 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:04.572008    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30a9ce8-7798-4ae1-8012-1d820e55234b-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.176555 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.176530    2561 generic.go:356] "Generic (PLEG): container finished" podID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerID="ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee" exitCode=0
Sep 17 19:13:05.177000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.176560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerDied","Data":"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"}
Sep 17 19:13:05.177000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.176579    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"
Sep 17 19:13:05.177000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.176594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz" event={"ID":"f30a9ce8-7798-4ae1-8012-1d820e55234b","Type":"ContainerDied","Data":"55bd53565581ef56853921c6551b45b3adf8820550d2b1b9fed1a09e5e2c5e56"}
Sep 17 19:13:05.177000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.176616    2561 scope.go:117] "RemoveContainer" containerID="ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"
Sep 17 19:13:05.185262 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.185246    2561 scope.go:117] "RemoveContainer" containerID="99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"
Sep 17 19:13:05.193391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.193370    2561 scope.go:117] "RemoveContainer" containerID="ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"
Sep 17 19:13:05.193659 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:13:05.193639    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee\": container with ID starting with ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee not found: ID does not exist" containerID="ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"
Sep 17 19:13:05.193744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.193669    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee"} err="failed to get container status \"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee\": rpc error: code = NotFound desc = could not find container \"ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee\": container with ID starting with ec49687bc1e1c5c40a574d645b93c079c42690d8d3e823bbb113dbdf4eaf10ee not found: ID does not exist"
Sep 17 19:13:05.193744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.193691    2561 scope.go:117] "RemoveContainer" containerID="99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"
Sep 17 19:13:05.194152 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:13:05.194125    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398\": container with ID starting with 99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398 not found: ID does not exist" containerID="99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"
Sep 17 19:13:05.194262 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.194169    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398"} err="failed to get container status \"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398\": rpc error: code = NotFound desc = could not find container \"99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398\": container with ID starting with 99bb341419bd11f13cf1bc3ef6c69a7256211dff8ae7dea1ace5b463bb889398 not found: ID does not exist"
Sep 17 19:13:05.195370 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.195349    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:13:05.199414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:05.199393    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6685f8446-dhvlz"]
Sep 17 19:13:07.086739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:07.086703    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" path="/var/lib/kubelet/pods/f30a9ce8-7798-4ae1-8012-1d820e55234b/volumes"
Sep 17 19:13:10.588764 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.588731    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:13:10.589240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.589091    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="storage-initializer"
Sep 17 19:13:10.589240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.589104    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="storage-initializer"
Sep 17 19:13:10.589240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.589122    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main"
Sep 17 19:13:10.589240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.589131    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main"
Sep 17 19:13:10.589240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.589211    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30a9ce8-7798-4ae1-8012-1d820e55234b" containerName="main"
Sep 17 19:13:10.592307 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.592281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.595730 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.595705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 19:13:10.595863 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.595706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:13:10.595863 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.595706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x7zbj\""
Sep 17 19:13:10.595863 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.595772    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:13:10.604301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.604282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:13:10.627230 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.627230 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.627394 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.627394 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg24t\" (UniqueName: \"kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.627394 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.627394 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.627381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728286 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg24t\" (UniqueName: \"kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728752 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728811 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.728811 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.728775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.730853 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.730832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.731020 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.731005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.737240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.737215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg24t\" (UniqueName: \"kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:10.903121 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:10.903039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:11.175858 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:11.175781    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:13:11.178780 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:13:11.178754    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba90188_6ce2_4ba9_8ef1_4a7c5292b627.slice/crio-57a86e6e46e32dddaaf892984f90a5e90f971f69b37676fbd3b01ae7058d48f8 WatchSource:0}: Error finding container 57a86e6e46e32dddaaf892984f90a5e90f971f69b37676fbd3b01ae7058d48f8: Status 404 returned error can't find the container with id 57a86e6e46e32dddaaf892984f90a5e90f971f69b37676fbd3b01ae7058d48f8
Sep 17 19:13:11.203566 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:11.203538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" event={"ID":"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627","Type":"ContainerStarted","Data":"57a86e6e46e32dddaaf892984f90a5e90f971f69b37676fbd3b01ae7058d48f8"}
Sep 17 19:13:11.450280 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:11.450193    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:11.450434 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:11.450319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:13:12.209889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:12.209844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" event={"ID":"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627","Type":"ContainerStarted","Data":"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"}
Sep 17 19:13:12.236139 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:12.236090    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" podStartSLOduration=2.236073177 podStartE2EDuration="2.236073177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:12.235027362 +0000 UTC m=+2009.376453585" watchObservedRunningTime="2026-09-17 19:13:12.236073177 +0000 UTC m=+2009.377499425"
Sep 17 19:13:12.903621 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:12.903585    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:20.904205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:20.904169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:20.917007 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:20.916985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:21.259739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:21.259648    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:13:21.450003 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:21.449971    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:31.449623 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:31.449585    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:34.027396 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.027326    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:13:34.133420 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.133386    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs\") pod \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") "
Sep 17 19:13:34.133585 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.133490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tpcg\" (UniqueName: \"kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg\") pod \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\" (UID: \"144342a2-d87f-433f-bf3e-5a0cf3ca150a\") "
Sep 17 19:13:34.135794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.135765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "144342a2-d87f-433f-bf3e-5a0cf3ca150a" (UID: "144342a2-d87f-433f-bf3e-5a0cf3ca150a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:34.135794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.135784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg" (OuterVolumeSpecName: "kube-api-access-6tpcg") pod "144342a2-d87f-433f-bf3e-5a0cf3ca150a" (UID: "144342a2-d87f-433f-bf3e-5a0cf3ca150a"). InnerVolumeSpecName "kube-api-access-6tpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:34.235103 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.235060    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144342a2-d87f-433f-bf3e-5a0cf3ca150a-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:34.235103 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.235099    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpcg\" (UniqueName: \"kubernetes.io/projected/144342a2-d87f-433f-bf3e-5a0cf3ca150a-kube-api-access-6tpcg\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:13:34.312094 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.312020    2561 generic.go:356] "Generic (PLEG): container finished" podID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerID="3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6" exitCode=0
Sep 17 19:13:34.312094 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.312068    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"
Sep 17 19:13:34.312266 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.312110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" event={"ID":"144342a2-d87f-433f-bf3e-5a0cf3ca150a","Type":"ContainerDied","Data":"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"}
Sep 17 19:13:34.312266 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.312170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns" event={"ID":"144342a2-d87f-433f-bf3e-5a0cf3ca150a","Type":"ContainerDied","Data":"d4ee41ba330809160efe064000360d85518c67465c756889c0834ee1972bd91a"}
Sep 17 19:13:34.312266 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.312196    2561 scope.go:117] "RemoveContainer" containerID="3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"
Sep 17 19:13:34.321909 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.321877    2561 scope.go:117] "RemoveContainer" containerID="3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"
Sep 17 19:13:34.322240 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:13:34.322205    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6\": container with ID starting with 3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6 not found: ID does not exist" containerID="3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"
Sep 17 19:13:34.322311 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.322248    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6"} err="failed to get container status \"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6\": rpc error: code = NotFound desc = could not find container \"3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6\": container with ID starting with 3858b12e25857e9f675c2ffb57cba9292c383d794f290c5ca1be119655be4ef6 not found: ID does not exist"
Sep 17 19:13:34.334022 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.333990    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:13:34.340405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:34.340370    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecg6ns"]
Sep 17 19:13:35.086808 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:35.086771    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" path="/var/lib/kubelet/pods/144342a2-d87f-433f-bf3e-5a0cf3ca150a/volumes"
Sep 17 19:13:41.629679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.629645    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:13:41.630122 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.629974    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main"
Sep 17 19:13:41.630122 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.629985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main"
Sep 17 19:13:41.630122 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.630049    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="144342a2-d87f-433f-bf3e-5a0cf3ca150a" containerName="main"
Sep 17 19:13:41.632744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.632723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.639744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.639722    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 19:13:41.639836 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.639765    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-vvw4j\""
Sep 17 19:13:41.639836 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.639722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-n86gz\""
Sep 17 19:13:41.639836 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.639818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 19:13:41.641347 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.641334    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 19:13:41.654759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.654736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:13:41.704057 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.704026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.704206 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.704101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2krl\" (UniqueName: \"kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.805335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.805300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2krl\" (UniqueName: \"kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.805473 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.805352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.807998 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.807979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.822993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.822971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2krl\" (UniqueName: \"kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:41.943033 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:41.942940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:42.251203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:42.251164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:13:42.253159 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:13:42.253121    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc5060_ff9e_40b0_9b80_e6615558ee00.slice/crio-d1c3430cf5981c455ebc94072f615572c94140e9b0b732eb62255023106265d5 WatchSource:0}: Error finding container d1c3430cf5981c455ebc94072f615572c94140e9b0b732eb62255023106265d5: Status 404 returned error can't find the container with id d1c3430cf5981c455ebc94072f615572c94140e9b0b732eb62255023106265d5
Sep 17 19:13:42.352005 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:42.351973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" event={"ID":"9adc5060-ff9e-40b0-9b80-e6615558ee00","Type":"ContainerStarted","Data":"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"}
Sep 17 19:13:42.352146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:42.352026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" event={"ID":"9adc5060-ff9e-40b0-9b80-e6615558ee00","Type":"ContainerStarted","Data":"d1c3430cf5981c455ebc94072f615572c94140e9b0b732eb62255023106265d5"}
Sep 17 19:13:42.352146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:42.352109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:13:42.376067 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:42.376021    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podStartSLOduration=1.376004876 podStartE2EDuration="1.376004876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:42.375369441 +0000 UTC m=+2039.516795656" watchObservedRunningTime="2026-09-17 19:13:42.376004876 +0000 UTC m=+2039.517431085"
Sep 17 19:13:59.796456 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:59.796420    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:13:59.805836 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:13:59.805802    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:13:59.805836 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:13:59.805836 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:13:59.805836 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979" cmd=["/bin/sleep","15"]
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:59.805851    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" message="" err=<
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:13:59.805865    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerName="main" containerID="cri-o://0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"
Sep 17 19:13:59.806119 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:13:59.805932    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerName="main" containerID="cri-o://0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979" gracePeriod=60
Sep 17 19:14:00.160317 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.160289    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:14:00.258144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258292 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg24t\" (UniqueName: \"kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258292 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258292 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258292 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache\") pod \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\" (UID: \"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627\") "
Sep 17 19:14:00.258520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:00.258520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home" (OuterVolumeSpecName: "home") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:00.258665 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258624    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.258665 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258649    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.258665 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.258647    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:00.260508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.260484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t" (OuterVolumeSpecName: "kube-api-access-kg24t") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "kube-api-access-kg24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:00.260956 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.260933    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm" (OuterVolumeSpecName: "dshm") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:00.260956 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.260941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" (UID: "9ba90188-6ce2-4ba9-8ef1-4a7c5292b627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:00.359305 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.359277    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.359305 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.359300    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.359496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.359312    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.359496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.359329    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg24t\" (UniqueName: \"kubernetes.io/projected/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627-kube-api-access-kg24t\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:14:00.435643 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.435609    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerID="0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979" exitCode=0
Sep 17 19:14:00.435790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.435658    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"
Sep 17 19:14:00.435790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.435689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" event={"ID":"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627","Type":"ContainerDied","Data":"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"}
Sep 17 19:14:00.435790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.435743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t" event={"ID":"9ba90188-6ce2-4ba9-8ef1-4a7c5292b627","Type":"ContainerDied","Data":"57a86e6e46e32dddaaf892984f90a5e90f971f69b37676fbd3b01ae7058d48f8"}
Sep 17 19:14:00.435790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.435768    2561 scope.go:117] "RemoveContainer" containerID="0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"
Sep 17 19:14:00.445195 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.445174    2561 scope.go:117] "RemoveContainer" containerID="0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"
Sep 17 19:14:00.445458 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:14:00.445436    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979\": container with ID starting with 0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979 not found: ID does not exist" containerID="0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"
Sep 17 19:14:00.445508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.445466    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979"} err="failed to get container status \"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979\": rpc error: code = NotFound desc = could not find container \"0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979\": container with ID starting with 0c66fb8617917b74c6010039193859e5a7746bbda93759a7f4dcff516d3da979 not found: ID does not exist"
Sep 17 19:14:00.461006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.460982    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:14:00.466829 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:00.466809    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-ghf4t"]
Sep 17 19:14:01.086223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:01.086194    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" path="/var/lib/kubelet/pods/9ba90188-6ce2-4ba9-8ef1-4a7c5292b627/volumes"
Sep 17 19:14:13.359801 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:13.359767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:14:45.109590 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:45.109564    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:14:45.128943 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:14:45.128922    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:15:51.525170 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.525062    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:15:51.525650 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.525466    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerName="main"
Sep 17 19:15:51.525650 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.525480    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerName="main"
Sep 17 19:15:51.525650 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.525555    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba90188-6ce2-4ba9-8ef1-4a7c5292b627" containerName="main"
Sep 17 19:15:51.528534 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.528515    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.531484 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.531458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:15:51.531600 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.531570    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-nn965\""
Sep 17 19:15:51.531820 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.531803    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 19:15:51.531914 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.531882    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 19:15:51.532066 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.532046    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 19:15:51.532126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.532115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 19:15:51.545790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.545770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:15:51.584719 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.584853 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.584853 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.584853 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.585026 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gsd\" (UniqueName: \"kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.585026 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.585026 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.584985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.585152 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.585057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686084 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gsd\" (UniqueName: \"kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686718 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686689    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686909 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.686909 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.687058 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.687058 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.686971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.688958 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.688938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.689153 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.689132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.695400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.695377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gsd\" (UniqueName: \"kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:51.839448 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:51.839412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:15:52.125139 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:52.125055    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:15:52.128029 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:15:52.128001    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63487210_d97a_40d4_a8cd_ff35eecc6b25.slice/crio-cdad65132929f81c402d4ba34a97bd4fefc21a6637bb3e38547b6469b37319dc WatchSource:0}: Error finding container cdad65132929f81c402d4ba34a97bd4fefc21a6637bb3e38547b6469b37319dc: Status 404 returned error can't find the container with id cdad65132929f81c402d4ba34a97bd4fefc21a6637bb3e38547b6469b37319dc
Sep 17 19:15:52.917036 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:52.916996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerStarted","Data":"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"}
Sep 17 19:15:52.917472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:52.917064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerStarted","Data":"cdad65132929f81c402d4ba34a97bd4fefc21a6637bb3e38547b6469b37319dc"}
Sep 17 19:15:55.933600 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:55.933564    2561 generic.go:356] "Generic (PLEG): container finished" podID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerID="c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af" exitCode=0
Sep 17 19:15:55.934065 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:55.933640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerDied","Data":"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"}
Sep 17 19:15:56.940650 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:56.940615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerStarted","Data":"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"}
Sep 17 19:15:56.968236 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:56.968179    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podStartSLOduration=2.917678804 podStartE2EDuration="5.96815829s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.050479486s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:56.96612334 +0000 UTC m=+2174.107549566" watchObservedRunningTime="2026-09-17 19:15:56.96815829 +0000 UTC m=+2174.109584502"
Sep 17 19:15:57.840432 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:15:57.840400    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:16:01.839673 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:01.839632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:16:01.841190 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:01.841162    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:08.679126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:08.679088    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:16:11.839923 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:11.839859    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:13.359124 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:13.359083    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:21.840230 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:21.840176    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:23.359114 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:23.359071    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:23.704851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:23.704756    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" containerID="cri-o://f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749" gracePeriod=45
Sep 17 19:16:31.840222 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:31.840175    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:33.359739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:33.359694    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:33.360147 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:33.359811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:16:41.840387 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:41.840348    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:43.359587 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:43.359546    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:51.840288 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:51.840245    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:16:53.359214 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:53.359180    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:53.960724 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:53.960702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:16:54.057475 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.057378    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2krl\" (UniqueName: \"kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl\") pod \"9adc5060-ff9e-40b0-9b80-e6615558ee00\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") "
Sep 17 19:16:54.057475 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.057473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs\") pod \"9adc5060-ff9e-40b0-9b80-e6615558ee00\" (UID: \"9adc5060-ff9e-40b0-9b80-e6615558ee00\") "
Sep 17 19:16:54.061346 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.061300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9adc5060-ff9e-40b0-9b80-e6615558ee00" (UID: "9adc5060-ff9e-40b0-9b80-e6615558ee00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:54.061346 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.061308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl" (OuterVolumeSpecName: "kube-api-access-v2krl") pod "9adc5060-ff9e-40b0-9b80-e6615558ee00" (UID: "9adc5060-ff9e-40b0-9b80-e6615558ee00"). InnerVolumeSpecName "kube-api-access-v2krl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:54.158959 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.158922    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc5060-ff9e-40b0-9b80-e6615558ee00-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:16:54.159197 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.159182    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2krl\" (UniqueName: \"kubernetes.io/projected/9adc5060-ff9e-40b0-9b80-e6615558ee00-kube-api-access-v2krl\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:16:54.225531 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.225495    2561 generic.go:356] "Generic (PLEG): container finished" podID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerID="f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749" exitCode=0
Sep 17 19:16:54.225696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.225555    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"
Sep 17 19:16:54.225696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.225565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" event={"ID":"9adc5060-ff9e-40b0-9b80-e6615558ee00","Type":"ContainerDied","Data":"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"}
Sep 17 19:16:54.225696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.225616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x" event={"ID":"9adc5060-ff9e-40b0-9b80-e6615558ee00","Type":"ContainerDied","Data":"d1c3430cf5981c455ebc94072f615572c94140e9b0b732eb62255023106265d5"}
Sep 17 19:16:54.225696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.225639    2561 scope.go:117] "RemoveContainer" containerID="f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"
Sep 17 19:16:54.236110 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.236084    2561 scope.go:117] "RemoveContainer" containerID="f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"
Sep 17 19:16:54.236563 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:16:54.236535    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749\": container with ID starting with f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749 not found: ID does not exist" containerID="f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"
Sep 17 19:16:54.236672 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.236573    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749"} err="failed to get container status \"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749\": rpc error: code = NotFound desc = could not find container \"f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749\": container with ID starting with f401ced55f793830a9a81493a72392a19b467465274e166e1fe2ef96642be749 not found: ID does not exist"
Sep 17 19:16:54.252797 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.252765    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:16:54.257595 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:54.257559    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schewwd8x"]
Sep 17 19:16:55.088969 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:16:55.088872    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" path="/var/lib/kubelet/pods/9adc5060-ff9e-40b0-9b80-e6615558ee00/volumes"
Sep 17 19:17:01.840540 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:01.840497    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 19:17:11.822862 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.822824    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:17:11.823279 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.823149    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main"
Sep 17 19:17:11.823279 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.823161    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main"
Sep 17 19:17:11.823279 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.823250    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc5060-ff9e-40b0-9b80-e6615558ee00" containerName="main"
Sep 17 19:17:11.826117 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.826092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:11.834176 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.834150    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:17:11.835161 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.835140    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-rj94h\""
Sep 17 19:17:11.835434 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.835210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-mbwml\""
Sep 17 19:17:11.835534 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.835445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:17:11.835534 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.835519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:17:11.846190 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.846156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:17:11.851659 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.851638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:17:11.863552 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.863527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:17:11.912809 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.912777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdwp\" (UniqueName: \"kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:11.912997 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:11.912867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.013952 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.013923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.014151 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.013991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdwp\" (UniqueName: \"kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.016761 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.016724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.026327 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.026296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdwp\" (UniqueName: \"kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.136693 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.136661    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:12.415919 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:12.415866    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:17:12.418119 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:17:12.418094    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3aa2d7_7e26_4fc3_b867_56f97b0f7393.slice/crio-5f94cac56f18dd7b0b97f6cf2ca68602d8dad1ee79b605808e6f9b7f0002e6f2 WatchSource:0}: Error finding container 5f94cac56f18dd7b0b97f6cf2ca68602d8dad1ee79b605808e6f9b7f0002e6f2: Status 404 returned error can't find the container with id 5f94cac56f18dd7b0b97f6cf2ca68602d8dad1ee79b605808e6f9b7f0002e6f2
Sep 17 19:17:13.325153 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:13.325116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" event={"ID":"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393","Type":"ContainerStarted","Data":"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"}
Sep 17 19:17:13.325542 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:13.325175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" event={"ID":"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393","Type":"ContainerStarted","Data":"5f94cac56f18dd7b0b97f6cf2ca68602d8dad1ee79b605808e6f9b7f0002e6f2"}
Sep 17 19:17:13.325542 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:13.325210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:13.342713 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:13.342664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podStartSLOduration=2.342649152 podStartE2EDuration="2.342649152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:13.341460604 +0000 UTC m=+2250.482886825" watchObservedRunningTime="2026-09-17 19:17:13.342649152 +0000 UTC m=+2250.484075375"
Sep 17 19:17:44.331967 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:44.331939    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:17:47.774994 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:17:47.774954    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:18:02.798911 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:02.798849    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main" containerID="cri-o://e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79" gracePeriod=45
Sep 17 19:18:03.451994 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.451971    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:18:03.565461 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565461 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565461 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565721 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565480    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565721 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565721 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5gsd\" (UniqueName: \"kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565931 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache" (OuterVolumeSpecName: "model-cache") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.565931 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.565931 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:03.565931 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.565865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home\") pod \"63487210-d97a-40d4-a8cd-ff35eecc6b25\" (UID: \"63487210-d97a-40d4-a8cd-ff35eecc6b25\") "
Sep 17 19:18:03.566200 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.566178    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63487210-d97a-40d4-a8cd-ff35eecc6b25-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.566257 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.566208    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.566374 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.566314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home" (OuterVolumeSpecName: "home") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.568491 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.568449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm" (OuterVolumeSpecName: "dshm") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.568629 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.568489    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd" (OuterVolumeSpecName: "kube-api-access-j5gsd") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "kube-api-access-j5gsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:03.568629 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.568580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:03.570367 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.570340    2561 generic.go:356] "Generic (PLEG): container finished" podID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerID="e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79" exitCode=0
Sep 17 19:18:03.570499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.570399    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"
Sep 17 19:18:03.570499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.570422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerDied","Data":"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"}
Sep 17 19:18:03.570499 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.570477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h" event={"ID":"63487210-d97a-40d4-a8cd-ff35eecc6b25","Type":"ContainerDied","Data":"cdad65132929f81c402d4ba34a97bd4fefc21a6637bb3e38547b6469b37319dc"}
Sep 17 19:18:03.570660 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.570509    2561 scope.go:117] "RemoveContainer" containerID="e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"
Sep 17 19:18:03.578571 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.578547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.584953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.584937    2561 scope.go:117] "RemoveContainer" containerID="c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"
Sep 17 19:18:03.591436 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.591414    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63487210-d97a-40d4-a8cd-ff35eecc6b25" (UID: "63487210-d97a-40d4-a8cd-ff35eecc6b25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:03.593223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.593200    2561 scope.go:117] "RemoveContainer" containerID="e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"
Sep 17 19:18:03.593482 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:18:03.593463    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79\": container with ID starting with e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79 not found: ID does not exist" containerID="e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"
Sep 17 19:18:03.593580 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.593495    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79"} err="failed to get container status \"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79\": rpc error: code = NotFound desc = could not find container \"e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79\": container with ID starting with e347386c9d573a2f626caf4721b68cf1fcde23aa26d03f6b7d7d86f8c6c11f79 not found: ID does not exist"
Sep 17 19:18:03.593580 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.593513    2561 scope.go:117] "RemoveContainer" containerID="c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"
Sep 17 19:18:03.593815 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:18:03.593796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af\": container with ID starting with c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af not found: ID does not exist" containerID="c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"
Sep 17 19:18:03.593863 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.593821    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af"} err="failed to get container status \"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af\": rpc error: code = NotFound desc = could not find container \"c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af\": container with ID starting with c646b79c156560a559d0b671ec45d50d71be76b9524a0289f53007438cb747af not found: ID does not exist"
Sep 17 19:18:03.667083 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667061    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63487210-d97a-40d4-a8cd-ff35eecc6b25-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.667083 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667081    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.667205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667091    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gsd\" (UniqueName: \"kubernetes.io/projected/63487210-d97a-40d4-a8cd-ff35eecc6b25-kube-api-access-j5gsd\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.667205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667099    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.667205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667107    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.667205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.667114    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63487210-d97a-40d4-a8cd-ff35eecc6b25-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:18:03.894819 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.894793    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:18:03.901108 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:03.901087    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dc85b49bb-dw48h"]
Sep 17 19:18:05.086511 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:05.086479    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" path="/var/lib/kubelet/pods/63487210-d97a-40d4-a8cd-ff35eecc6b25/volumes"
Sep 17 19:18:58.020855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.020816    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:18:58.021304 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.021167    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="storage-initializer"
Sep 17 19:18:58.021304 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.021179    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="storage-initializer"
Sep 17 19:18:58.021304 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.021191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main"
Sep 17 19:18:58.021304 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.021197    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main"
Sep 17 19:18:58.021304 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.021255    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="63487210-d97a-40d4-a8cd-ff35eecc6b25" containerName="main"
Sep 17 19:18:58.025413 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.025385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.028402 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.028366    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 19:18:58.029170 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.029150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-b5scb\""
Sep 17 19:18:58.029247 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.029160    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:18:58.029345 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.029321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:18:58.029617 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.029602    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:18:58.029665 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.029620    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:18:58.043014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.042987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:18:58.119308 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknmd\" (UniqueName: \"kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.119502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.119489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.220739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknmd\" (UniqueName: \"kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221006 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.220966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221442 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221442 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221535 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.221535 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.221481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.222126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.222100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.223451 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.223423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.223803 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.223783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.231963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.231942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknmd\" (UniqueName: \"kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.336856 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.336816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:18:58.672335 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:18:58.672301    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1807c674_fe5d_4baf_bdeb_b1d53e9ed1d3.slice/crio-03dfaece5e7da9f26d399897165b35cef26991f18419de653986d85fe8f9356b WatchSource:0}: Error finding container 03dfaece5e7da9f26d399897165b35cef26991f18419de653986d85fe8f9356b: Status 404 returned error can't find the container with id 03dfaece5e7da9f26d399897165b35cef26991f18419de653986d85fe8f9356b
Sep 17 19:18:58.674121 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.674099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:18:58.819553 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.819517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerStarted","Data":"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"}
Sep 17 19:18:58.819733 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:18:58.819580    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerStarted","Data":"03dfaece5e7da9f26d399897165b35cef26991f18419de653986d85fe8f9356b"}
Sep 17 19:19:02.842443 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:02.842404    2561 generic.go:356] "Generic (PLEG): container finished" podID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerID="a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604" exitCode=0
Sep 17 19:19:02.842871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:02.842470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerDied","Data":"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"}
Sep 17 19:19:03.849518 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:03.849481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerStarted","Data":"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"}
Sep 17 19:19:03.873668 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:03.873605    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podStartSLOduration=3.7359598590000003 podStartE2EDuration="6.873584926s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.137625067s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:03.87179407 +0000 UTC m=+2361.013220282" watchObservedRunningTime="2026-09-17 19:19:03.873584926 +0000 UTC m=+2361.015011139"
Sep 17 19:19:04.337838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:04.337802    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:19:08.337979 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:08.337935    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:19:08.339943 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:08.339889    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:19:16.724865 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:16.724810    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:19:18.337705 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:18.337653    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:19:24.331258 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:24.331216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:28.338280 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:28.338229    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:19:31.746955 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:31.746879    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main" containerID="cri-o://073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f" gracePeriod=45
Sep 17 19:19:34.332158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:34.332114    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:38.338246 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:38.338205    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:19:44.331263 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:44.331224    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:44.331714 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:44.331350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:19:45.141085 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:45.140970    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:19:45.158881 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:45.158858    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:19:48.337685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:48.337637    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:19:54.331864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:54.331824    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:58.337967 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:19:58.337926    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:20:01.999030 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:01.998993    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:20:02.126613 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.126581    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs\") pod \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") "
Sep 17 19:20:02.126826 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.126646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzdwp\" (UniqueName: \"kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp\") pod \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\" (UID: \"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393\") "
Sep 17 19:20:02.129186 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.129149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp" (OuterVolumeSpecName: "kube-api-access-rzdwp") pod "2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" (UID: "2a3aa2d7-7e26-4fc3-b867-56f97b0f7393"). InnerVolumeSpecName "kube-api-access-rzdwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:02.129186 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.129159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" (UID: "2a3aa2d7-7e26-4fc3-b867-56f97b0f7393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:02.138681 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.138653    2561 generic.go:356] "Generic (PLEG): container finished" podID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerID="073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f" exitCode=0
Sep 17 19:20:02.138807 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.138704    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"
Sep 17 19:20:02.138807 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.138706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" event={"ID":"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393","Type":"ContainerDied","Data":"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"}
Sep 17 19:20:02.138948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.138834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62" event={"ID":"2a3aa2d7-7e26-4fc3-b867-56f97b0f7393","Type":"ContainerDied","Data":"5f94cac56f18dd7b0b97f6cf2ca68602d8dad1ee79b605808e6f9b7f0002e6f2"}
Sep 17 19:20:02.138948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.138868    2561 scope.go:117] "RemoveContainer" containerID="073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"
Sep 17 19:20:02.148030 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.148009    2561 scope.go:117] "RemoveContainer" containerID="073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"
Sep 17 19:20:02.148351 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:20:02.148330    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f\": container with ID starting with 073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f not found: ID does not exist" containerID="073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"
Sep 17 19:20:02.148427 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.148361    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f"} err="failed to get container status \"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f\": rpc error: code = NotFound desc = could not find container \"073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f\": container with ID starting with 073cd38d29299528a6e97a3f676d7222aa16186fda78629eb15079cf4e9fb10f not found: ID does not exist"
Sep 17 19:20:02.163294 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.163269    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:20:02.167649 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.167624    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc587zv62"]
Sep 17 19:20:02.228002 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.227965    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:20:02.228002 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:02.227996    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdwp\" (UniqueName: \"kubernetes.io/projected/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393-kube-api-access-rzdwp\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:20:03.086955 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:03.086919    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" path="/var/lib/kubelet/pods/2a3aa2d7-7e26-4fc3-b867-56f97b0f7393/volumes"
Sep 17 19:20:07.060957 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.060926    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:20:07.061439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.061420    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main"
Sep 17 19:20:07.061522 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.061442    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main"
Sep 17 19:20:07.061575 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.061534    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3aa2d7-7e26-4fc3-b867-56f97b0f7393" containerName="main"
Sep 17 19:20:07.066026 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.065992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.069395 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.069353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:20:07.069527 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.069403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-6hc2m\""
Sep 17 19:20:07.069527 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.069438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:20:07.069799 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.069782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:20:07.079414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.079387    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:20:07.175161 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s84\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.175550 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.175463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s84\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276870 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276870 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276870 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.276870 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277139 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.276930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277201 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.277180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277323 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.277294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277323 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.277306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277457 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.277339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.277457 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.277356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.279191 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.279167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.279790 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.279773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.286706 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.286661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.287262 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.287235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s84\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84\") pod \"router-gateway-1-openshift-default-cc7cb45dd-lq9nd\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.377003 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.376967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:07.659148 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.659121    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:20:07.662087 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:20:07.662054    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7468e00b_246d_4a44_98b2_51d60a7315b8.slice/crio-cb165bb86199b7fed6f8087ecc015b4634b82abb64e95fc7ef0feb3083dce667 WatchSource:0}: Error finding container cb165bb86199b7fed6f8087ecc015b4634b82abb64e95fc7ef0feb3083dce667: Status 404 returned error can't find the container with id cb165bb86199b7fed6f8087ecc015b4634b82abb64e95fc7ef0feb3083dce667
Sep 17 19:20:07.664486 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.664453    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:07.664570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.664521    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:07.664570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:07.664550    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:08.171324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.171289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerStarted","Data":"843a1b1f24fd9b058152ac2a50680776733d364076043ff2c9198df7b817402a"}
Sep 17 19:20:08.171832 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.171343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerStarted","Data":"cb165bb86199b7fed6f8087ecc015b4634b82abb64e95fc7ef0feb3083dce667"}
Sep 17 19:20:08.195946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.195869    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podStartSLOduration=1.19585282 podStartE2EDuration="1.19585282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:08.193669798 +0000 UTC m=+2425.335096010" watchObservedRunningTime="2026-09-17 19:20:08.19585282 +0000 UTC m=+2425.337279032"
Sep 17 19:20:08.337826 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.337783    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:20:08.377326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.377296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:08.382883 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:08.382856    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:09.176834 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:09.176795    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:09.177762 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:09.177744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:18.337595 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:18.337555    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 17 19:20:20.238312 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.238277    2561 generic.go:356] "Generic (PLEG): container finished" podID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerID="843a1b1f24fd9b058152ac2a50680776733d364076043ff2c9198df7b817402a" exitCode=0
Sep 17 19:20:20.238821 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.238352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerDied","Data":"843a1b1f24fd9b058152ac2a50680776733d364076043ff2c9198df7b817402a"}
Sep 17 19:20:20.238821 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.238764    2561 scope.go:117] "RemoveContainer" containerID="843a1b1f24fd9b058152ac2a50680776733d364076043ff2c9198df7b817402a"
Sep 17 19:20:20.239887 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.239854    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:20.239978 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.239965    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:20.240031 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:20.240014    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:20:21.245510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:21.245475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerStarted","Data":"ce4983fe045548c5772f76e27011292e94e52da034eb2335b883e1ca58751f69"}
Sep 17 19:20:21.377347 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:21.377303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:21.378738 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:21.378704    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:22.377759 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:22.377708    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:23.377948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:23.377886    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:24.177129 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:24.177087    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:24.377582 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:24.377528    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:25.378082 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:25.378033    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:26.377587 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:26.377544    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 19:20:27.381322 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:27.381291    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:27.382378 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:27.382355    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:20:28.348258 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:28.348221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:20:28.356268 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:28.356241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:20:46.257188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:20:46.257147    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:21:01.286149 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:01.286087    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main" containerID="cri-o://894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992" gracePeriod=45
Sep 17 19:21:01.929630 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:01.929597    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:21:02.082866 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.082834    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.082889    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.082939    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wknmd\" (UniqueName: \"kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.082964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083291 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083291 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083291 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache\") pod \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\" (UID: \"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3\") "
Sep 17 19:21:02.083291 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:02.083502 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083439    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:02.083561 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home" (OuterVolumeSpecName: "home") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:02.083561 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083519    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.083671 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.083567    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.084072 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.084036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:02.085659 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.085635    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:02.085886 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.085862    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm" (OuterVolumeSpecName: "dshm") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:02.086012 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.085880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd" (OuterVolumeSpecName: "kube-api-access-wknmd") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "kube-api-access-wknmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:02.109081 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.109049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" (UID: "1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:02.184872 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184840    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.185038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184930    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.185038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184954    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknmd\" (UniqueName: \"kubernetes.io/projected/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-kube-api-access-wknmd\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.185038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184970    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.185038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184983    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.185038 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.184991    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:21:02.446420 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.446333    2561 generic.go:356] "Generic (PLEG): container finished" podID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerID="894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992" exitCode=0
Sep 17 19:21:02.446420 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.446384    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"
Sep 17 19:21:02.446974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.446389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerDied","Data":"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"}
Sep 17 19:21:02.446974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.446524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854" event={"ID":"1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3","Type":"ContainerDied","Data":"03dfaece5e7da9f26d399897165b35cef26991f18419de653986d85fe8f9356b"}
Sep 17 19:21:02.446974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.446568    2561 scope.go:117] "RemoveContainer" containerID="894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"
Sep 17 19:21:02.458083 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.458065    2561 scope.go:117] "RemoveContainer" containerID="a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"
Sep 17 19:21:02.466851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.466834    2561 scope.go:117] "RemoveContainer" containerID="894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"
Sep 17 19:21:02.467273 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:21:02.467250    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992\": container with ID starting with 894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992 not found: ID does not exist" containerID="894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"
Sep 17 19:21:02.467391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.467284    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992"} err="failed to get container status \"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992\": rpc error: code = NotFound desc = could not find container \"894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992\": container with ID starting with 894e080b8a1fcedaf6c57ceea49efbabfe8cd361ff306eb0e96245fb7700d992 not found: ID does not exist"
Sep 17 19:21:02.467391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.467308    2561 scope.go:117] "RemoveContainer" containerID="a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"
Sep 17 19:21:02.467693 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:21:02.467661    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604\": container with ID starting with a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604 not found: ID does not exist" containerID="a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"
Sep 17 19:21:02.467786 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.467701    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604"} err="failed to get container status \"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604\": rpc error: code = NotFound desc = could not find container \"a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604\": container with ID starting with a9fff7b428b7ef8ce503897f2474043b6b2d5f027ea697c4f604824ffd7ac604 not found: ID does not exist"
Sep 17 19:21:02.469852 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.469832    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:21:02.473438 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:02.473411    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5bc7cfcfcc-mt854"]
Sep 17 19:21:03.087088 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:03.087057    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" path="/var/lib/kubelet/pods/1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3/volumes"
Sep 17 19:21:43.814252 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814157    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:21:43.814824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main"
Sep 17 19:21:43.814824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814539    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main"
Sep 17 19:21:43.814824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814548    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="storage-initializer"
Sep 17 19:21:43.814824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814554    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="storage-initializer"
Sep 17 19:21:43.814824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.814604    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1807c674-fe5d-4baf-bdeb-b1d53e9ed1d3" containerName="main"
Sep 17 19:21:43.816682 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.816661    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.819224 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.819192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 19:21:43.819367 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.819297    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:21:43.819461 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.819443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wpgxx\""
Sep 17 19:21:43.819533 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.819465    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:21:43.820042 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.820026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:21:43.820124 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.820035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:21:43.831555 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.831532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:21:43.950562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lf6j\" (UniqueName: \"kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:43.950961 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:43.950770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051555 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051716 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051716 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051716 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051716 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.051716 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052002 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052002 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.051754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf6j\" (UniqueName: \"kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052084 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.052007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052084 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.052033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052230 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.052211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052313 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.052292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.052559 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.052541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.054225 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.054204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.054589 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.054572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.060796 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.060772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lf6j\" (UniqueName: \"kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j\") pod \"stop-feature-test-kserve-8575df5f6c-6cjwf\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.128010 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.127982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:44.404326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.404300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:21:44.405557 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:21:44.405527    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e9ebd9_e9dd_4b1b_893b_76fff5f07a3f.slice/crio-aee2bf9f87ca330f0aade303aeeb03bb9f7faeefd8ae967e4655e6b7641ccf0d WatchSource:0}: Error finding container aee2bf9f87ca330f0aade303aeeb03bb9f7faeefd8ae967e4655e6b7641ccf0d: Status 404 returned error can't find the container with id aee2bf9f87ca330f0aade303aeeb03bb9f7faeefd8ae967e4655e6b7641ccf0d
Sep 17 19:21:44.634274 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.634239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerStarted","Data":"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"}
Sep 17 19:21:44.634478 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:44.634295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerStarted","Data":"aee2bf9f87ca330f0aade303aeeb03bb9f7faeefd8ae967e4655e6b7641ccf0d"}
Sep 17 19:21:47.656501 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:47.656465    2561 generic.go:356] "Generic (PLEG): container finished" podID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerID="153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871" exitCode=0
Sep 17 19:21:47.657004 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:47.656497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerDied","Data":"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"}
Sep 17 19:21:48.663573 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:48.663533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerStarted","Data":"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"}
Sep 17 19:21:48.691060 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:48.690918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podStartSLOduration=3.058226264 podStartE2EDuration="5.690873811s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.632647547s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:48.690370507 +0000 UTC m=+2525.831796720" watchObservedRunningTime="2026-09-17 19:21:48.690873811 +0000 UTC m=+2525.832300025"
Sep 17 19:21:49.128834 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:49.128794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:54.128614 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:54.128567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:21:54.129989 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:21:54.129963    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:04.128775 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:04.128728    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:14.129087 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:14.129027    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:24.129449 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:24.129401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:34.128831 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:34.128777    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:44.128457 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:44.128410    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:22:54.129324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:22:54.129275    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:04.128914 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:04.128853    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:14.129347 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:14.129296    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:24.129269 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:24.129227    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:32.007298 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:32.007258    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:23:32.007937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:32.007479    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy" containerID="cri-o://ce4983fe045548c5772f76e27011292e94e52da034eb2335b883e1ca58751f69" gracePeriod=30
Sep 17 19:23:34.128802 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:34.128734    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:37.185632 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.185602    2561 generic.go:356] "Generic (PLEG): container finished" podID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerID="ce4983fe045548c5772f76e27011292e94e52da034eb2335b883e1ca58751f69" exitCode=0
Sep 17 19:23:37.186086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.185644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerDied","Data":"ce4983fe045548c5772f76e27011292e94e52da034eb2335b883e1ca58751f69"}
Sep 17 19:23:37.186086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.185705    2561 scope.go:117] "RemoveContainer" containerID="843a1b1f24fd9b058152ac2a50680776733d364076043ff2c9198df7b817402a"
Sep 17 19:23:37.268780 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.268708    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:23:37.416417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416386    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416467    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416512    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416567    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2s84\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416624    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.416685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.417016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token\") pod \"7468e00b-246d-4a44-98b2-51d60a7315b8\" (UID: \"7468e00b-246d-4a44-98b2-51d60a7315b8\") "
Sep 17 19:23:37.417016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:37.417016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.416872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:37.417171 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.417090    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.417171 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.417110    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7468e00b-246d-4a44-98b2-51d60a7315b8-istiod-ca-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.417285 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.417167    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:37.417285 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.417209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:37.417285 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.417251    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:37.419414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.419367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84" (OuterVolumeSpecName: "kube-api-access-h2s84") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "kube-api-access-h2s84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:37.419414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.419368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:23:37.419612 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.419469    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:37.419612 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.419547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7468e00b-246d-4a44-98b2-51d60a7315b8" (UID: "7468e00b-246d-4a44-98b2-51d60a7315b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:37.518574 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518530    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-envoy\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518574 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518577    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-credential-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518595    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-workload-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518609    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2s84\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-kube-api-access-h2s84\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518622    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-data\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518637    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-podinfo\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:37.518812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:37.518649    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7468e00b-246d-4a44-98b2-51d60a7315b8-istio-token\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:23:38.192445 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:38.192411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"
Sep 17 19:23:38.192445 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:38.192411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd" event={"ID":"7468e00b-246d-4a44-98b2-51d60a7315b8","Type":"ContainerDied","Data":"cb165bb86199b7fed6f8087ecc015b4634b82abb64e95fc7ef0feb3083dce667"}
Sep 17 19:23:38.193029 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:38.192476    2561 scope.go:117] "RemoveContainer" containerID="ce4983fe045548c5772f76e27011292e94e52da034eb2335b883e1ca58751f69"
Sep 17 19:23:38.219556 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:38.219522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:23:38.224431 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:38.224404    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-lq9nd"]
Sep 17 19:23:39.087260 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:39.087227    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" path="/var/lib/kubelet/pods/7468e00b-246d-4a44-98b2-51d60a7315b8/volumes"
Sep 17 19:23:44.129079 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:44.129029    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 19:23:50.478391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478356    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:23:50.478841 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478800    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.478841 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478814    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.478841 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478827    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.478841 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478833    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.479035 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478914    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.479035 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.478924    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7468e00b-246d-4a44-98b2-51d60a7315b8" containerName="istio-proxy"
Sep 17 19:23:50.483086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.483068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.486705 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.486682    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:23:50.487045 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.487021    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:23:50.487188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.487020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-l2qb5\""
Sep 17 19:23:50.487188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.487090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-qppnr\""
Sep 17 19:23:50.487188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.487127    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:23:50.494821 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.494799    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:23:50.635373 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.635338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89t7\" (UniqueName: \"kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.635548 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.635401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.736113 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.736027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89t7\" (UniqueName: \"kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.736113 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.736093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.738631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.738603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.746308 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.746283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89t7\" (UniqueName: \"kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:50.793779 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:50.793745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:51.077194 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:51.077168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:23:51.079099 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:23:51.079069    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b169caa_1f41_432c_93a8_ef290d06eb60.slice/crio-76f4750f6084c5f35295943610ce233716981bd2d51d4be9691a6a37e4438230 WatchSource:0}: Error finding container 76f4750f6084c5f35295943610ce233716981bd2d51d4be9691a6a37e4438230: Status 404 returned error can't find the container with id 76f4750f6084c5f35295943610ce233716981bd2d51d4be9691a6a37e4438230
Sep 17 19:23:51.253315 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:51.253270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" event={"ID":"3b169caa-1f41-432c-93a8-ef290d06eb60","Type":"ContainerStarted","Data":"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"}
Sep 17 19:23:51.253519 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:51.253325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" event={"ID":"3b169caa-1f41-432c-93a8-ef290d06eb60","Type":"ContainerStarted","Data":"76f4750f6084c5f35295943610ce233716981bd2d51d4be9691a6a37e4438230"}
Sep 17 19:23:51.253519 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:51.253398    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:23:51.274717 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:51.274666    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podStartSLOduration=1.274652031 podStartE2EDuration="1.274652031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:51.27282777 +0000 UTC m=+2648.414254007" watchObservedRunningTime="2026-09-17 19:23:51.274652031 +0000 UTC m=+2648.416078242"
Sep 17 19:23:54.138851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:54.138821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:23:54.147237 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:23:54.147215    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:24:03.401706 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:03.401672    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:24:17.799198 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.799167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:24:17.802525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.802504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.815164 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.815140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:24:17.864198 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsrc\" (UniqueName: \"kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864756 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.864756 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.864393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965645 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965645 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.965929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.965832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsrc\" (UniqueName: \"kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.966301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.966143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.966301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.966284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.966424 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.966388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.966491 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.966442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.966849 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.966827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.968526 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.968500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.969025 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.969003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:17.979929 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:17.979910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsrc\" (UniqueName: \"kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc\") pod \"stop-feature-test-kserve-8575df5f6c-w2fpp\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:18.113551 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:18.113521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:18.397787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:18.397709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:24:18.400563 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:24:18.400535    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf924e90b_5f48_4f6b_9e64_83fe466495a6.slice/crio-f172896524fc5bc3c0eca1ce52cea4888758e5cb4424a856b90fd47dc498bb4b WatchSource:0}: Error finding container f172896524fc5bc3c0eca1ce52cea4888758e5cb4424a856b90fd47dc498bb4b: Status 404 returned error can't find the container with id f172896524fc5bc3c0eca1ce52cea4888758e5cb4424a856b90fd47dc498bb4b
Sep 17 19:24:18.425039 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:18.424980    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" containerID="cri-o://bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574" gracePeriod=45
Sep 17 19:24:19.198947 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.198922    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:24:19.280417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280505    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280530    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lf6j\" (UniqueName: \"kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs\") pod \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\" (UID: \"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f\") "
Sep 17 19:24:19.280982 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:19.281042 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.280979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:19.281106 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.281050    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home" (OuterVolumeSpecName: "home") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:19.281181 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.281161    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.281240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.281185    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.281240 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.281202    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.286951 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.284544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:19.286951 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.284572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm" (OuterVolumeSpecName: "dshm") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:19.286951 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.286285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j" (OuterVolumeSpecName: "kube-api-access-7lf6j") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "kube-api-access-7lf6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:19.297765 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.297740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:19.308063 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.308040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" (UID: "66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:19.379682 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.379649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerStarted","Data":"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"}
Sep 17 19:24:19.379865 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.379714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerStarted","Data":"f172896524fc5bc3c0eca1ce52cea4888758e5cb4424a856b90fd47dc498bb4b"}
Sep 17 19:24:19.381411 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381382    2561 generic.go:356] "Generic (PLEG): container finished" podID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerID="bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574" exitCode=0
Sep 17 19:24:19.381525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerDied","Data":"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"}
Sep 17 19:24:19.381525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"
Sep 17 19:24:19.381525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" event={"ID":"66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f","Type":"ContainerDied","Data":"aee2bf9f87ca330f0aade303aeeb03bb9f7faeefd8ae967e4655e6b7641ccf0d"}
Sep 17 19:24:19.381525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381495    2561 scope.go:117] "RemoveContainer" containerID="bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"
Sep 17 19:24:19.381760 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381743    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.381824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381778    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lf6j\" (UniqueName: \"kubernetes.io/projected/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kube-api-access-7lf6j\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.381824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381793    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.381824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381807    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.381824 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.381820    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:24:19.390596 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.390515    2561 scope.go:117] "RemoveContainer" containerID="153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"
Sep 17 19:24:19.398454 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.398436    2561 scope.go:117] "RemoveContainer" containerID="bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"
Sep 17 19:24:19.398766 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:24:19.398738    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574\": container with ID starting with bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574 not found: ID does not exist" containerID="bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"
Sep 17 19:24:19.398838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.398780    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574"} err="failed to get container status \"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574\": rpc error: code = NotFound desc = could not find container \"bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574\": container with ID starting with bb1941000e6cfb470e94bfd40de54fc89a518c99e253396ff02adf456d582574 not found: ID does not exist"
Sep 17 19:24:19.398838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.398805    2561 scope.go:117] "RemoveContainer" containerID="153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"
Sep 17 19:24:19.399400 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:24:19.399263    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871\": container with ID starting with 153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871 not found: ID does not exist" containerID="153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"
Sep 17 19:24:19.399400 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.399297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871"} err="failed to get container status \"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871\": rpc error: code = NotFound desc = could not find container \"153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871\": container with ID starting with 153db933e92d859e6aa1c39e89d227e31d590ed5e49c1677c02492bd5d1f9871 not found: ID does not exist"
Sep 17 19:24:19.413441 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.413415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:24:19.417501 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:19.417476    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf"]
Sep 17 19:24:20.140397 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:20.140345    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-6cjwf" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:24:21.086775 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:21.086747    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" path="/var/lib/kubelet/pods/66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f/volumes"
Sep 17 19:24:21.393503 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:21.393415    2561 generic.go:356] "Generic (PLEG): container finished" podID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerID="735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709" exitCode=0
Sep 17 19:24:21.393503 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:21.393489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerDied","Data":"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"}
Sep 17 19:24:22.259125 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:22.259096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:24:22.400094 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:22.400064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerStarted","Data":"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"}
Sep 17 19:24:22.424243 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:22.424180    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podStartSLOduration=3.137794466 podStartE2EDuration="5.424158206s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.28636374s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:22.420619144 +0000 UTC m=+2679.562045368" watchObservedRunningTime="2026-09-17 19:24:22.424158206 +0000 UTC m=+2679.565584421"
Sep 17 19:24:23.114493 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:23.114460    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:28.114593 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:28.114490    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:24:28.115777 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:28.115752    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:24:38.114075 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:38.114035    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:24:45.172693 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:45.172667    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:24:45.187573 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:45.187546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:24:48.114287 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:48.114247    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:24:58.114671 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:24:58.114629    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:08.114821 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:08.114774    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:18.114758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:18.114718    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:28.114217 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:28.114170    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:38.114059 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:38.114022    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:48.114527 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:48.114487    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:25:58.114483 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:25:58.114440    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:26:08.114497 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:08.114450    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 19:26:18.128616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:18.128580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:26:18.137214 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:18.137183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:26:20.068241 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:20.068194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:26:35.097766 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:35.097720    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main" containerID="cri-o://391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045" gracePeriod=45
Sep 17 19:26:35.956973 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:35.956945    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:26:36.024306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024270    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nsrc\" (UniqueName: \"kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024725 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024725 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024725 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024886 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs\") pod \"f924e90b-5f48-4f6b-9e64-83fe466495a6\" (UID: \"f924e90b-5f48-4f6b-9e64-83fe466495a6\") "
Sep 17 19:26:36.024886 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024776    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:36.025037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home" (OuterVolumeSpecName: "home") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:36.025037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.024981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:36.025223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.025194    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.025223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.025214    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f924e90b-5f48-4f6b-9e64-83fe466495a6-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.025337 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.025229    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.027273 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.027246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:36.027273 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.027256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm" (OuterVolumeSpecName: "dshm") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:36.027452 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.027415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc" (OuterVolumeSpecName: "kube-api-access-8nsrc") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "kube-api-access-8nsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:36.042535 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.042488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:36.044875 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.044843    2561 generic.go:356] "Generic (PLEG): container finished" podID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerID="391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045" exitCode=0
Sep 17 19:26:36.045385 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.044930    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"
Sep 17 19:26:36.045385 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.044890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerDied","Data":"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"}
Sep 17 19:26:36.045385 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.045014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp" event={"ID":"f924e90b-5f48-4f6b-9e64-83fe466495a6","Type":"ContainerDied","Data":"f172896524fc5bc3c0eca1ce52cea4888758e5cb4424a856b90fd47dc498bb4b"}
Sep 17 19:26:36.045385 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.045037    2561 scope.go:117] "RemoveContainer" containerID="391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"
Sep 17 19:26:36.054518 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.054495    2561 scope.go:117] "RemoveContainer" containerID="735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"
Sep 17 19:26:36.059339 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.059301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f924e90b-5f48-4f6b-9e64-83fe466495a6" (UID: "f924e90b-5f48-4f6b-9e64-83fe466495a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:36.063583 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.063562    2561 scope.go:117] "RemoveContainer" containerID="391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"
Sep 17 19:26:36.063921 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:26:36.063881    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045\": container with ID starting with 391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045 not found: ID does not exist" containerID="391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"
Sep 17 19:26:36.064007 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.063936    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045"} err="failed to get container status \"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045\": rpc error: code = NotFound desc = could not find container \"391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045\": container with ID starting with 391eba70491aef7f4bd6d1a2a8059a15f1684d0281146bbe1a97ef892b15d045 not found: ID does not exist"
Sep 17 19:26:36.064007 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.063965    2561 scope.go:117] "RemoveContainer" containerID="735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"
Sep 17 19:26:36.064290 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:26:36.064267    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709\": container with ID starting with 735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709 not found: ID does not exist" containerID="735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"
Sep 17 19:26:36.064331 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.064297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709"} err="failed to get container status \"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709\": rpc error: code = NotFound desc = could not find container \"735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709\": container with ID starting with 735a19e6390bca38e30fce8ae69294bdd5f362a77dce05f9c9b23e0665300709 not found: ID does not exist"
Sep 17 19:26:36.126148 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.126106    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.126148 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.126140    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f924e90b-5f48-4f6b-9e64-83fe466495a6-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.126148 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.126150    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.126624 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.126160    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsrc\" (UniqueName: \"kubernetes.io/projected/f924e90b-5f48-4f6b-9e64-83fe466495a6-kube-api-access-8nsrc\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.126624 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.126171    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f924e90b-5f48-4f6b-9e64-83fe466495a6-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:26:36.368628 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.368586    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:26:36.371699 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:36.371666    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-8575df5f6c-w2fpp"]
Sep 17 19:26:37.087289 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:37.087252    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" path="/var/lib/kubelet/pods/f924e90b-5f48-4f6b-9e64-83fe466495a6/volumes"
Sep 17 19:26:50.290739 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:50.290702    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:26:52.259533 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:26:52.259488    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:02.259312 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:02.259269    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:05.311917 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:05.311837    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" containerID="cri-o://84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164" gracePeriod=45
Sep 17 19:27:12.259728 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:12.259688    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:12.260214 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:12.259807    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:27:22.259950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:22.259885    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:32.259382 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:32.259291    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:35.563474 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.563445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:27:35.657378 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.657345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r89t7\" (UniqueName: \"kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7\") pod \"3b169caa-1f41-432c-93a8-ef290d06eb60\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") "
Sep 17 19:27:35.657378 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.657380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs\") pod \"3b169caa-1f41-432c-93a8-ef290d06eb60\" (UID: \"3b169caa-1f41-432c-93a8-ef290d06eb60\") "
Sep 17 19:27:35.659843 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.659803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b169caa-1f41-432c-93a8-ef290d06eb60" (UID: "3b169caa-1f41-432c-93a8-ef290d06eb60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:35.659987 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.659883    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7" (OuterVolumeSpecName: "kube-api-access-r89t7") pod "3b169caa-1f41-432c-93a8-ef290d06eb60" (UID: "3b169caa-1f41-432c-93a8-ef290d06eb60"). InnerVolumeSpecName "kube-api-access-r89t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:35.758058 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.758029    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89t7\" (UniqueName: \"kubernetes.io/projected/3b169caa-1f41-432c-93a8-ef290d06eb60-kube-api-access-r89t7\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.758058 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:35.758052    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b169caa-1f41-432c-93a8-ef290d06eb60-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:27:36.320222 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.320180    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerID="84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164" exitCode=0
Sep 17 19:27:36.320407 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.320249    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"
Sep 17 19:27:36.320407 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.320261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" event={"ID":"3b169caa-1f41-432c-93a8-ef290d06eb60","Type":"ContainerDied","Data":"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"}
Sep 17 19:27:36.320407 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.320323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46" event={"ID":"3b169caa-1f41-432c-93a8-ef290d06eb60","Type":"ContainerDied","Data":"76f4750f6084c5f35295943610ce233716981bd2d51d4be9691a6a37e4438230"}
Sep 17 19:27:36.320407 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.320345    2561 scope.go:117] "RemoveContainer" containerID="84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"
Sep 17 19:27:36.329666 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.329647    2561 scope.go:117] "RemoveContainer" containerID="84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"
Sep 17 19:27:36.329981 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:27:36.329962    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164\": container with ID starting with 84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164 not found: ID does not exist" containerID="84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"
Sep 17 19:27:36.330045 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.329990    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164"} err="failed to get container status \"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164\": rpc error: code = NotFound desc = could not find container \"84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164\": container with ID starting with 84842640d3def5c0768251baaeec0b99a4d2438cca11fc6e8331688dcb4da164 not found: ID does not exist"
Sep 17 19:27:36.344115 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.344086    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:27:36.347363 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:36.347339    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schedsl46"]
Sep 17 19:27:37.086957 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:37.086923    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" path="/var/lib/kubelet/pods/3b169caa-1f41-432c-93a8-ef290d06eb60/volumes"
Sep 17 19:27:47.533334 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533295    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="storage-initializer"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533639    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="storage-initializer"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533655    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533663    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533668    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533680    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="storage-initializer"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533686    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="storage-initializer"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533692    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533697    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main"
Sep 17 19:27:47.533753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f924e90b-5f48-4f6b-9e64-83fe466495a6" containerName="main"
Sep 17 19:27:47.534175 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533763    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b169caa-1f41-432c-93a8-ef290d06eb60" containerName="main"
Sep 17 19:27:47.534175 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.533771    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e9ebd9-e9dd-4b1b-893b-76fff5f07a3f" containerName="main"
Sep 17 19:27:47.538000 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.537977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.540556 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.540534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-5zmls\""
Sep 17 19:27:47.540942 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.540918    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:27:47.541071 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.540951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:27:47.541071 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.540958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:27:47.541071 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.540919    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-8mnn5\""
Sep 17 19:27:47.541464 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.541447    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:27:47.543361 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.543342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:27:47.569686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.569651    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:27:47.664313 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlzr\" (UniqueName: \"kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.664544 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.664546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.765983 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.765929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.765991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlzr\" (UniqueName: \"kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766439 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766657 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766657 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766732 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.766944 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.766921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.768581 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.768558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.768787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.768768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.784349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.784263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlzr\" (UniqueName: \"kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr\") pod \"custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:47.848949 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:47.848888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:48.136218 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:48.136190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:27:48.139093 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:27:48.139066    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268af241_88b3_4b66_88e3_663158285ee1.slice/crio-852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75 WatchSource:0}: Error finding container 852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75: Status 404 returned error can't find the container with id 852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75
Sep 17 19:27:48.374014 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:48.373977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerStarted","Data":"b4fd3b125649e88d7f6a2b35acab5ef2aab3f2bb92f7f56ba7c173a50c0eddbe"}
Sep 17 19:27:48.374203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:48.374031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerStarted","Data":"852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75"}
Sep 17 19:27:48.374203 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:48.374116    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:49.381350 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:49.381304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerStarted","Data":"7a979930a57cd32f478df48d1747626c96711a607f6d5e30b99c73cb9d54f6ae"}
Sep 17 19:27:51.391479 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:51.391444    2561 generic.go:356] "Generic (PLEG): container finished" podID="268af241-88b3-4b66-88e3-663158285ee1" containerID="7a979930a57cd32f478df48d1747626c96711a607f6d5e30b99c73cb9d54f6ae" exitCode=0
Sep 17 19:27:51.391965 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:51.391510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerDied","Data":"7a979930a57cd32f478df48d1747626c96711a607f6d5e30b99c73cb9d54f6ae"}
Sep 17 19:27:52.398281 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:52.398245    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerStarted","Data":"45a04f686f8a0b7b70f6936a97217464fe63819eacc48e42a191d0de49535869"}
Sep 17 19:27:52.436430 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:52.436353    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podStartSLOduration=2.952871865 podStartE2EDuration="5.436329477s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.483457612s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:52.434125561 +0000 UTC m=+2889.575551772" watchObservedRunningTime="2026-09-17 19:27:52.436329477 +0000 UTC m=+2889.577755689"
Sep 17 19:27:57.849440 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:57.849395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:57.850262 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:57.849481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:27:57.851018 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:27:57.850984    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:07.850155 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:07.850101    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:07.866182 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:07.866154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:28:17.850391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:17.850343    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:27.849631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:27.849586    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:37.849946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:37.849867    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:47.849410 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:47.849354    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:28:57.849915 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:28:57.849854    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:29:07.850519 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:07.850183    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:29:17.850232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:17.850189    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:29:27.849379 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:27.849335    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:29:37.849553 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:37.849503    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8001/health\": dial tcp 10.134.0.50:8001: connect: connection refused"
Sep 17 19:29:45.203673 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:45.203644    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:29:45.215023 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:45.214999    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:29:47.859846 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:47.859806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:29:47.876170 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:29:47.876140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:30:12.247727 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:12.247686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:30:27.271215 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:27.271118    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main" containerID="cri-o://45a04f686f8a0b7b70f6936a97217464fe63819eacc48e42a191d0de49535869" gracePeriod=45
Sep 17 19:30:27.889079 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:27.889044    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="llm-d-routing-sidecar" containerID="cri-o://b4fd3b125649e88d7f6a2b35acab5ef2aab3f2bb92f7f56ba7c173a50c0eddbe" gracePeriod=45
Sep 17 19:30:28.134681 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134653    2561 generic.go:356] "Generic (PLEG): container finished" podID="268af241-88b3-4b66-88e3-663158285ee1" containerID="45a04f686f8a0b7b70f6936a97217464fe63819eacc48e42a191d0de49535869" exitCode=0
Sep 17 19:30:28.134681 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134677    2561 generic.go:356] "Generic (PLEG): container finished" podID="268af241-88b3-4b66-88e3-663158285ee1" containerID="b4fd3b125649e88d7f6a2b35acab5ef2aab3f2bb92f7f56ba7c173a50c0eddbe" exitCode=0
Sep 17 19:30:28.134933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerDied","Data":"45a04f686f8a0b7b70f6936a97217464fe63819eacc48e42a191d0de49535869"}
Sep 17 19:30:28.134933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerDied","Data":"b4fd3b125649e88d7f6a2b35acab5ef2aab3f2bb92f7f56ba7c173a50c0eddbe"}
Sep 17 19:30:28.134933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5" event={"ID":"268af241-88b3-4b66-88e3-663158285ee1","Type":"ContainerDied","Data":"852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75"}
Sep 17 19:30:28.134933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.134803    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852f14b41aad89e13226e8cf1824cb2820262583a9ef16f69bc22c7a95dcef75"
Sep 17 19:30:28.135142 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.135018    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:30:28.216602 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.216787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216626    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.216787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.216787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.216787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.216787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.217078 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216798    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.217078 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.216847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhlzr\" (UniqueName: \"kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr\") pod \"268af241-88b3-4b66-88e3-663158285ee1\" (UID: \"268af241-88b3-4b66-88e3-663158285ee1\") "
Sep 17 19:30:28.217078 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.217026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:28.217078 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.217026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache" (OuterVolumeSpecName: "model-cache") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.218355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.217425    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home" (OuterVolumeSpecName: "home") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.218355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.217592    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.218355 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.218281    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268af241-88b3-4b66-88e3-663158285ee1-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.220113 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.220065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm" (OuterVolumeSpecName: "dshm") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.220247 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.220212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr" (OuterVolumeSpecName: "kube-api-access-lhlzr") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "kube-api-access-lhlzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:28.220247 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.220234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:28.231693 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.231667    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.242105 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.242082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "268af241-88b3-4b66-88e3-663158285ee1" (UID: "268af241-88b3-4b66-88e3-663158285ee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318784    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318807    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/268af241-88b3-4b66-88e3-663158285ee1-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318816    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318825    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlzr\" (UniqueName: \"kubernetes.io/projected/268af241-88b3-4b66-88e3-663158285ee1-kube-api-access-lhlzr\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318834    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.318851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:28.318842    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/268af241-88b3-4b66-88e3-663158285ee1-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:30:29.138661 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:29.138630    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"
Sep 17 19:30:29.157631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:29.157598    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:30:29.161647 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:29.161617    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b4676b97d-zb2p5"]
Sep 17 19:30:31.087055 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:30:31.087025    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268af241-88b3-4b66-88e3-663158285ee1" path="/var/lib/kubelet/pods/268af241-88b3-4b66-88e3-663158285ee1/volumes"
Sep 17 19:31:02.468857 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.468819    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:31:02.469338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469319    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main"
Sep 17 19:31:02.469405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469339    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main"
Sep 17 19:31:02.469405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469355    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="storage-initializer"
Sep 17 19:31:02.469405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469365    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="storage-initializer"
Sep 17 19:31:02.469405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469389    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="llm-d-routing-sidecar"
Sep 17 19:31:02.469405 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469398    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="llm-d-routing-sidecar"
Sep 17 19:31:02.469602 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469478    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="main"
Sep 17 19:31:02.469602 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.469495    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="268af241-88b3-4b66-88e3-663158285ee1" containerName="llm-d-routing-sidecar"
Sep 17 19:31:02.472834 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.472816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.476935 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.476907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:31:02.477072 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.477024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:31:02.477151 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.477067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:31:02.477151 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.477071    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-lqvzz\""
Sep 17 19:31:02.494205 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.494185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:31:02.512780 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.512883 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.512883 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.512978 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.512978 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.512978 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvbj\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.513073 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.512999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.513073 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.513047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.513177 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.513092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614364 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614545 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614545 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614642 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614642 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614642 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvbj\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614796 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614796 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614796 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614991 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614991 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.614991 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.614978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.615154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.615133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.615273 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.615251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.617049 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.617028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.617367 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.617351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.625601 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.625573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvbj\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.626697 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.626673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-7h6bp\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:02.784832 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:02.784741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:03.074289 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.074145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:31:03.077117 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:31:03.077086    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2616f5a4_b6ea_46ee_ac84_c4318fc14712.slice/crio-77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162 WatchSource:0}: Error finding container 77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162: Status 404 returned error can't find the container with id 77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162
Sep 17 19:31:03.079468 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.079430    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:03.079572 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.079495    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:03.079572 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.079524    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:03.286661 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.286628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerStarted","Data":"b721b752a0ef4a26fb70eb42f418b168e48efdfe74840b5480c17e1af9fa3010"}
Sep 17 19:31:03.286832 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.286680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerStarted","Data":"77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162"}
Sep 17 19:31:03.313615 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.313562    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podStartSLOduration=1.31354486 podStartE2EDuration="1.31354486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:03.312094242 +0000 UTC m=+3080.453520468" watchObservedRunningTime="2026-09-17 19:31:03.31354486 +0000 UTC m=+3080.454971072"
Sep 17 19:31:03.785634 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:03.785598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:04.790686 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:04.790657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:05.296165 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:05.296127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:05.297065 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:05.297046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:11.447718 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.447678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:31:11.452182 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.452147    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.454696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.454664    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:31:11.454817 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.454768    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:31:11.454918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.454746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:31:11.454918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.454711    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-zxhpq\""
Sep 17 19:31:11.455051 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.454757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lwnjk\""
Sep 17 19:31:11.467310 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.467284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:31:11.595086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.594999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595086 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5qk\" (UniqueName: \"kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.595335 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.595324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696585 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696585 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5qk\" (UniqueName: \"kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.696825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697112 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.696843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697112 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.697064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.697133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697223 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.697187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697297 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.697228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.697582 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.697561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.699331 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.699306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.699832 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.699815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.705792 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.705769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5qk\" (UniqueName: \"kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk\") pod \"router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:11.765075 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:11.765033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:12.058126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:12.058082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:31:12.060941 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:31:12.060883    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac7d920_a3cc_4d8e_8beb_46aada5fa3ec.slice/crio-0b3dba0b007d2f797ecad1d6258f2319d8369e9c8d13136cada440f7526ea88b WatchSource:0}: Error finding container 0b3dba0b007d2f797ecad1d6258f2319d8369e9c8d13136cada440f7526ea88b: Status 404 returned error can't find the container with id 0b3dba0b007d2f797ecad1d6258f2319d8369e9c8d13136cada440f7526ea88b
Sep 17 19:31:12.332128 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:12.332088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerStarted","Data":"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"}
Sep 17 19:31:12.332318 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:12.332157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerStarted","Data":"0b3dba0b007d2f797ecad1d6258f2319d8369e9c8d13136cada440f7526ea88b"}
Sep 17 19:31:12.332318 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:12.332195    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:13.340881 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:13.340845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerStarted","Data":"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"}
Sep 17 19:31:15.361679 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.361644    2561 generic.go:356] "Generic (PLEG): container finished" podID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerID="b721b752a0ef4a26fb70eb42f418b168e48efdfe74840b5480c17e1af9fa3010" exitCode=0
Sep 17 19:31:15.362241 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.361715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerDied","Data":"b721b752a0ef4a26fb70eb42f418b168e48efdfe74840b5480c17e1af9fa3010"}
Sep 17 19:31:15.362241 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.362133    2561 scope.go:117] "RemoveContainer" containerID="b721b752a0ef4a26fb70eb42f418b168e48efdfe74840b5480c17e1af9fa3010"
Sep 17 19:31:15.363195 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.363156    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:15.363321 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.363249    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:15.363321 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.363298    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 19:31:15.363855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.363835    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerID="e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8" exitCode=0
Sep 17 19:31:15.363975 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:15.363917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerDied","Data":"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"}
Sep 17 19:31:16.373228 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:16.373189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerStarted","Data":"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"}
Sep 17 19:31:16.375531 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:16.375502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerStarted","Data":"0054ed63556eacede3ac57f21645dec0871b83d25a0fae1071b7785df39fb5ba"}
Sep 17 19:31:16.402087 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:16.401998    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podStartSLOduration=2.983344628 podStartE2EDuration="5.401983384s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.418638756s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:16.400103137 +0000 UTC m=+3093.541529350" watchObservedRunningTime="2026-09-17 19:31:16.401983384 +0000 UTC m=+3093.543409595"
Sep 17 19:31:16.785491 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:16.785396    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:16.786616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:16.786585    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 17 19:31:17.785885 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:17.785843    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 17 19:31:18.785758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:18.785710    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 17 19:31:19.785816 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:19.785762    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 17 19:31:20.296162 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:20.296121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:20.785960 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:20.785890    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 17 19:31:21.765446 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:21.765400    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:21.765643 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:21.765482    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:21.766980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:21.766952    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:31:21.789357 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:21.789325    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:21.790364 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:21.790345    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:31:31.765809 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:31.765759    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:31:31.779102 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:31.779073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:31:41.766348 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:41.766299    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:31:51.765409 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:31:51.765364    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:01.765934 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:01.765788    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:11.766399 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:11.766338    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:21.766428 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:21.766384    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:31.765427 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:31.765387    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:41.765423 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:41.765367    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:32:51.766458 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:32:51.766403    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:33:01.766543 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:01.766431    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8001/health\": dial tcp 10.134.0.52:8001: connect: connection refused"
Sep 17 19:33:11.775011 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:11.774973    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:33:11.787208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:11.787184    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:33:30.994140 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:30.994059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:33:46.017267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.017214    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main" containerID="cri-o://bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc" gracePeriod=45
Sep 17 19:33:46.635155 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.635117    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="llm-d-routing-sidecar" containerID="cri-o://bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9" gracePeriod=45
Sep 17 19:33:46.882486 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.882460    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:33:46.982780 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.982745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.982976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.982805    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.982976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.982830    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.982976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.982877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.982976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.982927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.983211 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983002    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.983211 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.983211 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r5qk\" (UniqueName: \"kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk\") pod \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\" (UID: \"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec\") "
Sep 17 19:33:46.983211 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:33:46.983466 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.983586 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983509    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.983794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.983767    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home" (OuterVolumeSpecName: "home") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.985629 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.985606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm" (OuterVolumeSpecName: "dshm") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.985769 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.985749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk" (OuterVolumeSpecName: "kube-api-access-7r5qk") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "kube-api-access-7r5qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:46.985834 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.985754    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:46.995159 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:46.995136    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:47.007092 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.007065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" (UID: "7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:47.084562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084541    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5qk\" (UniqueName: \"kubernetes.io/projected/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kube-api-access-7r5qk\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084562 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084562    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084572    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084581    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084589    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084597    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.084946 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.084604    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.111324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111296    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerID="bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc" exitCode=0
Sep 17 19:33:47.111324 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111318    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerID="bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9" exitCode=0
Sep 17 19:33:47.111494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerDied","Data":"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"}
Sep 17 19:33:47.111494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"
Sep 17 19:33:47.111494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerDied","Data":"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"}
Sep 17 19:33:47.111494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4" event={"ID":"7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec","Type":"ContainerDied","Data":"0b3dba0b007d2f797ecad1d6258f2319d8369e9c8d13136cada440f7526ea88b"}
Sep 17 19:33:47.111494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.111449    2561 scope.go:117] "RemoveContainer" containerID="bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"
Sep 17 19:33:47.119889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.119871    2561 scope.go:117] "RemoveContainer" containerID="e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"
Sep 17 19:33:47.127998 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.127980    2561 scope.go:117] "RemoveContainer" containerID="bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"
Sep 17 19:33:47.135268 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.135248    2561 scope.go:117] "RemoveContainer" containerID="bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"
Sep 17 19:33:47.135525 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:33:47.135507    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc\": container with ID starting with bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc not found: ID does not exist" containerID="bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"
Sep 17 19:33:47.135576 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.135533    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"} err="failed to get container status \"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc\": rpc error: code = NotFound desc = could not find container \"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc\": container with ID starting with bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc not found: ID does not exist"
Sep 17 19:33:47.135576 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.135549    2561 scope.go:117] "RemoveContainer" containerID="e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"
Sep 17 19:33:47.135776 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:33:47.135760    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8\": container with ID starting with e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8 not found: ID does not exist" containerID="e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"
Sep 17 19:33:47.135827 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.135780    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"} err="failed to get container status \"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8\": rpc error: code = NotFound desc = could not find container \"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8\": container with ID starting with e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8 not found: ID does not exist"
Sep 17 19:33:47.135827 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.135802    2561 scope.go:117] "RemoveContainer" containerID="bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"
Sep 17 19:33:47.136024 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:33:47.136007    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9\": container with ID starting with bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9 not found: ID does not exist" containerID="bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"
Sep 17 19:33:47.136076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136027    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"} err="failed to get container status \"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9\": rpc error: code = NotFound desc = could not find container \"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9\": container with ID starting with bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9 not found: ID does not exist"
Sep 17 19:33:47.136076 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136040    2561 scope.go:117] "RemoveContainer" containerID="bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"
Sep 17 19:33:47.136245 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136229    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc"} err="failed to get container status \"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc\": rpc error: code = NotFound desc = could not find container \"bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc\": container with ID starting with bab88245387c44750689599af612c230f5f59dacf8e1486bc92de77cbdd457fc not found: ID does not exist"
Sep 17 19:33:47.136290 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136245    2561 scope.go:117] "RemoveContainer" containerID="e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"
Sep 17 19:33:47.136501 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136479    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8"} err="failed to get container status \"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8\": rpc error: code = NotFound desc = could not find container \"e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8\": container with ID starting with e1dfa4342cf9e2f347a4a77d375f5224ce3f6efae91fb513f0415ef597e0e6a8 not found: ID does not exist"
Sep 17 19:33:47.136501 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136504    2561 scope.go:117] "RemoveContainer" containerID="bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"
Sep 17 19:33:47.136768 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.136746    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9"} err="failed to get container status \"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9\": rpc error: code = NotFound desc = could not find container \"bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9\": container with ID starting with bfccec1652cd0f67759bbdd7aea88d3f413da10becf0acaee55cc7ff6d7fa1a9 not found: ID does not exist"
Sep 17 19:33:47.153091 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.153064    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:33:47.160694 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:47.160674    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-569b9f4f7b-kxrq4"]
Sep 17 19:33:49.088344 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:33:49.088313    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" path="/var/lib/kubelet/pods/7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec/volumes"
Sep 17 19:34:26.064779 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:26.064730    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:34:26.065425 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:26.065039    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy" containerID="cri-o://0054ed63556eacede3ac57f21645dec0871b83d25a0fae1071b7785df39fb5ba" gracePeriod=30
Sep 17 19:34:31.320520 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.320484    2561 generic.go:356] "Generic (PLEG): container finished" podID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerID="0054ed63556eacede3ac57f21645dec0871b83d25a0fae1071b7785df39fb5ba" exitCode=0
Sep 17 19:34:31.321016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.320539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerDied","Data":"0054ed63556eacede3ac57f21645dec0871b83d25a0fae1071b7785df39fb5ba"}
Sep 17 19:34:31.321016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.320592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp" event={"ID":"2616f5a4-b6ea-46ee-ac84-c4318fc14712","Type":"ContainerDied","Data":"77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162"}
Sep 17 19:34:31.321016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.320610    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e56500ecf0e68bca9bdec47eaf87c38a45a923dc47572be66f61646cb6d162"
Sep 17 19:34:31.321016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.320624    2561 scope.go:117] "RemoveContainer" containerID="b721b752a0ef4a26fb70eb42f418b168e48efdfe74840b5480c17e1af9fa3010"
Sep 17 19:34:31.324677 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.324594    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:34:31.381738 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.381963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.381963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llvbj\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.381963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.381963 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.382232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.381974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.382232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.382232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.382232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo\") pod \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\" (UID: \"2616f5a4-b6ea-46ee-ac84-c4318fc14712\") "
Sep 17 19:34:31.382232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382129    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:31.382510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:31.382510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data" (OuterVolumeSpecName: "istio-data") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:31.382510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:31.382510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382492    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.382510 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382511    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-data\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.382768 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382525    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-workload-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.382768 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382541    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-credential-socket\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.382877 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.382844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:31.385094 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.385045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj" (OuterVolumeSpecName: "kube-api-access-llvbj") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "kube-api-access-llvbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:31.385276 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.385172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:34:31.385354 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.385274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:31.385417 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.385351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token" (OuterVolumeSpecName: "istio-token") pod "2616f5a4-b6ea-46ee-ac84-c4318fc14712" (UID: "2616f5a4-b6ea-46ee-ac84-c4318fc14712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:31.483275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.483233    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-token\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.483275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.483269    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-podinfo\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.483275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.483279    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llvbj\" (UniqueName: \"kubernetes.io/projected/2616f5a4-b6ea-46ee-ac84-c4318fc14712-kube-api-access-llvbj\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.483547 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.483289    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istiod-ca-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:31.483547 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:31.483299    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616f5a4-b6ea-46ee-ac84-c4318fc14712-istio-envoy\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:34:32.330911 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:32.330861    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"
Sep 17 19:34:32.363914 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:32.363852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:34:32.368209 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:32.368174    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-7h6bp"]
Sep 17 19:34:33.087696 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:33.087655    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" path="/var/lib/kubelet/pods/2616f5a4-b6ea-46ee-ac84-c4318fc14712/volumes"
Sep 17 19:34:41.422133 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422096    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422462    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422492    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="llm-d-routing-sidecar"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422532    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="llm-d-routing-sidecar"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422539    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="storage-initializer"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422545    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="storage-initializer"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422607    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="llm-d-routing-sidecar"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422616    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.422616 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422623    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac7d920-a3cc-4d8e-8beb-46aada5fa3ec" containerName="main"
Sep 17 19:34:41.423053 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422678    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.423053 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422684    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.423053 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.422741    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2616f5a4-b6ea-46ee-ac84-c4318fc14712" containerName="istio-proxy"
Sep 17 19:34:41.427015 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.426989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.430077 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430053    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 19:34:41.430256 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430109    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 19:34:41.430256 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430182    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:34:41.430389 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430303    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 19:34:41.430389 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 19:34:41.430389 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.430356    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-8tws4\""
Sep 17 19:34:41.442663 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.442633    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:34:41.473444 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.473740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.473740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.473740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.473740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdjz\" (UniqueName: \"kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.473740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.474093 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.474093 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.473819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574529 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574529 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdjz\" (UniqueName: \"kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.574838 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.574751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.575201 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.575128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.575265 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.575242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.575321 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.575294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.575378 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.575349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.575478 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.575436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.577302 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.577281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.577426 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.577413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.583980 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.583949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdjz\" (UniqueName: \"kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:41.738177 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:41.738070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:42.022030 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:42.021994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:34:42.381236 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:42.381180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerStarted","Data":"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"}
Sep 17 19:34:42.381479 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:42.381260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerStarted","Data":"ff4ed34da2090329b86e7bbac8f6879e19d75ce9f15a84b26bb7dacfe1fa09d5"}
Sep 17 19:34:45.242354 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.242309    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:34:45.250493 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.250465    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:34:45.398591 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.398552    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerID="c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86" exitCode=0
Sep 17 19:34:45.398785 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.398624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerDied","Data":"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"}
Sep 17 19:34:45.412108 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.412074    2561 scope.go:117] "RemoveContainer" containerID="45a04f686f8a0b7b70f6936a97217464fe63819eacc48e42a191d0de49535869"
Sep 17 19:34:45.427072 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.427049    2561 scope.go:117] "RemoveContainer" containerID="b4fd3b125649e88d7f6a2b35acab5ef2aab3f2bb92f7f56ba7c173a50c0eddbe"
Sep 17 19:34:45.439298 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:45.439270    2561 scope.go:117] "RemoveContainer" containerID="7a979930a57cd32f478df48d1747626c96711a607f6d5e30b99c73cb9d54f6ae"
Sep 17 19:34:46.406778 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:46.406736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerStarted","Data":"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"}
Sep 17 19:34:46.432871 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:46.432803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podStartSLOduration=2.763645827 podStartE2EDuration="5.432780264s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.669134437s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:46.428460524 +0000 UTC m=+3303.569886736" watchObservedRunningTime="2026-09-17 19:34:46.432780264 +0000 UTC m=+3303.574206478"
Sep 17 19:34:46.738479 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:46.738369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:51.738788 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:51.738736    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:34:51.740536 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:34:51.740499    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:01.738812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:01.738756    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:11.739152 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:11.739099    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:21.739088 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:21.739017    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:31.738855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:31.738798    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:41.739342 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:41.739295    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:35:51.739673 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:35:51.739616    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:01.738617 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:01.738570    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:11.738993 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:11.738949    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:21.739126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:21.739073    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:31.738736 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:31.738636    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:41.739487 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:41.739430    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:36:51.739074 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:36:51.739016    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:37:01.749140 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:01.749037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:37:01.757079 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:01.757050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:37:06.698235 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:06.698194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:37:21.722613 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:21.722566    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main" containerID="cri-o://27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508" gracePeriod=45
Sep 17 19:37:22.577729 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.577697    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:37:22.609189 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609189 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jdjz\" (UniqueName: \"kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609370    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.609737 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.609457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache\") pod \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\" (UID: \"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4\") "
Sep 17 19:37:22.610525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.610147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:22.610525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.610218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:22.610525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.610259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home" (OuterVolumeSpecName: "home") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:22.612496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.612464    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:22.613969 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.613936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm" (OuterVolumeSpecName: "dshm") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:22.614369 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.614331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz" (OuterVolumeSpecName: "kube-api-access-5jdjz") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "kube-api-access-5jdjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:22.625350 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.625315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:22.639620 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.639554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" (UID: "b5e64988-2541-4e4e-93c2-8eb3e4dc70e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:22.711062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711001    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711055    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711062 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711066    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711077    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711088    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711098    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdjz\" (UniqueName: \"kubernetes.io/projected/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kube-api-access-5jdjz\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711112    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:22.711326 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:22.711123    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:37:23.207456 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.207421    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerID="27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508" exitCode=0
Sep 17 19:37:23.207918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.207497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerDied","Data":"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"}
Sep 17 19:37:23.207918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.207556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5" event={"ID":"b5e64988-2541-4e4e-93c2-8eb3e4dc70e4","Type":"ContainerDied","Data":"ff4ed34da2090329b86e7bbac8f6879e19d75ce9f15a84b26bb7dacfe1fa09d5"}
Sep 17 19:37:23.207918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.207559    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"
Sep 17 19:37:23.207918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.207582    2561 scope.go:117] "RemoveContainer" containerID="27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"
Sep 17 19:37:23.224267 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.224237    2561 scope.go:117] "RemoveContainer" containerID="c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"
Sep 17 19:37:23.228843 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.228811    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:37:23.233044 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.233012    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-64464c95f9fdkw5"]
Sep 17 19:37:23.235242 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.235221    2561 scope.go:117] "RemoveContainer" containerID="27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"
Sep 17 19:37:23.235580 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:37:23.235552    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508\": container with ID starting with 27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508 not found: ID does not exist" containerID="27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"
Sep 17 19:37:23.235653 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.235591    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508"} err="failed to get container status \"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508\": rpc error: code = NotFound desc = could not find container \"27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508\": container with ID starting with 27b714d0443160cf939a2f73efa0edad24805a190e087edfcb850df89a581508 not found: ID does not exist"
Sep 17 19:37:23.235653 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.235613    2561 scope.go:117] "RemoveContainer" containerID="c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"
Sep 17 19:37:23.235946 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:37:23.235929    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86\": container with ID starting with c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86 not found: ID does not exist" containerID="c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"
Sep 17 19:37:23.236002 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:23.235953    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86"} err="failed to get container status \"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86\": rpc error: code = NotFound desc = could not find container \"c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86\": container with ID starting with c2ed8a97d7e9a3bf2737b507f84ab1bfda469aae7da8c427393a522a39368e86 not found: ID does not exist"
Sep 17 19:37:25.088032 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:25.087990    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" path="/var/lib/kubelet/pods/b5e64988-2541-4e4e-93c2-8eb3e4dc70e4/volumes"
Sep 17 19:37:40.511492 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511431    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:37:40.512019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511826    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="storage-initializer"
Sep 17 19:37:40.512019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="storage-initializer"
Sep 17 19:37:40.512019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511861    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main"
Sep 17 19:37:40.512019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511866    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main"
Sep 17 19:37:40.512019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.511959    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e64988-2541-4e4e-93c2-8eb3e4dc70e4" containerName="main"
Sep 17 19:37:40.516079 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.516055    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.519190 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.519164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 19:37:40.519799 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.519776    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 19:37:40.519855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.519815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 19:37:40.520151 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.520136    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 19:37:40.520671 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.520655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:37:40.520762 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.520742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-c2th7\""
Sep 17 19:37:40.520818 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.520782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-qbhvm\""
Sep 17 19:37:40.531525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.531493    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:37:40.674647 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674647 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8ng\" (UniqueName: \"kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.674976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.674934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.775914 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.775795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.775914 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.775840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.775920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.775949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.775971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776158 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8ng\" (UniqueName: \"kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776464 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776464 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776464 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776592 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.776828 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.776809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.778541 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.778519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.778665 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.778626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.791522 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.791486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8ng\" (UniqueName: \"kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:40.828722 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:40.828681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:41.137618 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:41.137580    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:37:41.141476 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:37:41.141443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d4be3f_b406_4d4b_aa90_04577f2e36e9.slice/crio-6f12b6602fce4f99dbe99ec114e39a3c7c7362472158e8626292150daa2a45d4 WatchSource:0}: Error finding container 6f12b6602fce4f99dbe99ec114e39a3c7c7362472158e8626292150daa2a45d4: Status 404 returned error can't find the container with id 6f12b6602fce4f99dbe99ec114e39a3c7c7362472158e8626292150daa2a45d4
Sep 17 19:37:41.301289 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:41.301246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerStarted","Data":"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"}
Sep 17 19:37:41.301490 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:41.301319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerStarted","Data":"6f12b6602fce4f99dbe99ec114e39a3c7c7362472158e8626292150daa2a45d4"}
Sep 17 19:37:44.318039 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:44.317932    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerID="d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d" exitCode=0
Sep 17 19:37:44.318039 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:44.317987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerDied","Data":"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"}
Sep 17 19:37:45.325460 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:45.325415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerStarted","Data":"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"}
Sep 17 19:37:45.351438 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:45.351353    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.690121863 podStartE2EDuration="5.351309221s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.661187358s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:45.349385252 +0000 UTC m=+3482.490811466" watchObservedRunningTime="2026-09-17 19:37:45.351309221 +0000 UTC m=+3482.492735433"
Sep 17 19:37:45.483579 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:45.483532    2561 scope.go:117] "RemoveContainer" containerID="0054ed63556eacede3ac57f21645dec0871b83d25a0fae1071b7785df39fb5ba"
Sep 17 19:37:45.828913 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:45.828862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:50.829968 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:50.829911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:37:50.831830 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:37:50.831796    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:00.829351 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:00.829305    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:10.829858 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:10.829809    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:20.829458 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:20.829394    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:29.417415 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.417369    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:38:29.421429 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.421403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.425425 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.425392    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 19:38:29.426346 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.426320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 19:38:29.426538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.426393    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:38:29.426538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.426514    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 19:38:29.426538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.426525    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ltq47\""
Sep 17 19:38:29.426766 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.426526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 19:38:29.437092 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.437059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:38:29.541877 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.541843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.541877 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.541880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.541962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.542015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.542070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.542112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.542157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.542363 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.542183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75jk\" (UniqueName: \"kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.642980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643284 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643488 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643488 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75jk\" (UniqueName: \"kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643488 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.643778 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.643753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.644099 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.644051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.644099 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.644068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.645316 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.645293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.646451 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.646418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.646558 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.646541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.656882 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.656853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75jk\" (UniqueName: \"kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk\") pod \"tls-verification-test-kserve-cd94bd5f-nhh6x\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:29.735884 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:29.735776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:30.020126 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:30.020098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:38:30.022785 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:38:30.022756    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62466d12_173d_452b_a412_c17de257c57a.slice/crio-6ad5a87ca1fa7d9fd8c5a3658ca84cf4104f42c895a23128b5989f36d463de5e WatchSource:0}: Error finding container 6ad5a87ca1fa7d9fd8c5a3658ca84cf4104f42c895a23128b5989f36d463de5e: Status 404 returned error can't find the container with id 6ad5a87ca1fa7d9fd8c5a3658ca84cf4104f42c895a23128b5989f36d463de5e
Sep 17 19:38:30.540191 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:30.540147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerStarted","Data":"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"}
Sep 17 19:38:30.540706 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:30.540220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerStarted","Data":"6ad5a87ca1fa7d9fd8c5a3658ca84cf4104f42c895a23128b5989f36d463de5e"}
Sep 17 19:38:30.829957 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:30.829851    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:33.560382 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:33.560346    2561 generic.go:356] "Generic (PLEG): container finished" podID="62466d12-173d-452b-a412-c17de257c57a" containerID="0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262" exitCode=0
Sep 17 19:38:33.560794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:33.560430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerDied","Data":"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"}
Sep 17 19:38:34.567485 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:34.567444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerStarted","Data":"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"}
Sep 17 19:38:34.594879 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:34.594795    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podStartSLOduration=2.849697665 podStartE2EDuration="5.594770797s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.745073132s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:38:34.591288011 +0000 UTC m=+3531.732714226" watchObservedRunningTime="2026-09-17 19:38:34.594770797 +0000 UTC m=+3531.736197009"
Sep 17 19:38:34.736506 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:34.736448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:39.736628 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:39.736582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:38:39.738449 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:39.738415    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:38:40.829710 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:40.829655    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:49.736643 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:49.736583    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:38:50.829328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:50.829282    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:38:59.736794 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:38:59.736736    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:00.829468 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:00.829391    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:39:09.736975 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:09.736915    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:10.830105 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:10.830052    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:39:19.737349 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:19.737288    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:20.829621 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:20.829576    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:39:29.736656 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:29.736525    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:30.839419 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:30.839383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:39:30.847808 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:30.847784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:39:37.297586 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:37.297544    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:39:39.736648 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:39.736604    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:45.275258 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:45.275231    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:39:45.280603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:45.280581    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:39:49.737005 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:49.736956    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:39:52.327603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:52.327561    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" containerID="cri-o://0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40" gracePeriod=45
Sep 17 19:39:52.840287 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:52.840238    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 19:39:53.378063 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.378015    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:39:53.474232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474133    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs8ng\" (UniqueName: \"kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474232 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home\") pod \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\" (UID: \"f1d4be3f-b406-4d4b-aa90-04577f2e36e9\") "
Sep 17 19:39:53.474768 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474676    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:53.474768 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.474760    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.475277 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.475256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home" (OuterVolumeSpecName: "home") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:53.475587 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.475568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:39:53.477397 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.477362    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm" (OuterVolumeSpecName: "dshm") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:53.477524 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.477444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:39:53.477839 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.477811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng" (OuterVolumeSpecName: "kube-api-access-xs8ng") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "kube-api-access-xs8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:39:53.487617 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.487589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:53.499451 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.499407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1d4be3f-b406-4d4b-aa90-04577f2e36e9" (UID: "f1d4be3f-b406-4d4b-aa90-04577f2e36e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:39:53.576348 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576304    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576352    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs8ng\" (UniqueName: \"kubernetes.io/projected/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kube-api-access-xs8ng\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576372    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576387    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576400    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576412    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.576508 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.576424    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d4be3f-b406-4d4b-aa90-04577f2e36e9-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:39:53.980955 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.980916    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerID="0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40" exitCode=0
Sep 17 19:39:53.981144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.980986    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:39:53.981144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.981001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerDied","Data":"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"}
Sep 17 19:39:53.981144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.981058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f1d4be3f-b406-4d4b-aa90-04577f2e36e9","Type":"ContainerDied","Data":"6f12b6602fce4f99dbe99ec114e39a3c7c7362472158e8626292150daa2a45d4"}
Sep 17 19:39:53.981144 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.981083    2561 scope.go:117] "RemoveContainer" containerID="0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"
Sep 17 19:39:53.990098 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.990073    2561 scope.go:117] "RemoveContainer" containerID="d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"
Sep 17 19:39:53.999272 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.999254    2561 scope.go:117] "RemoveContainer" containerID="0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"
Sep 17 19:39:53.999538 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:39:53.999517    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40\": container with ID starting with 0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40 not found: ID does not exist" containerID="0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"
Sep 17 19:39:53.999611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.999552    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40"} err="failed to get container status \"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40\": rpc error: code = NotFound desc = could not find container \"0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40\": container with ID starting with 0ae0c1188ff19214b29c59b3d29e3bfbf3671ce5d5ecbb4012dd7ce4c347eb40 not found: ID does not exist"
Sep 17 19:39:53.999611 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.999577    2561 scope.go:117] "RemoveContainer" containerID="d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"
Sep 17 19:39:53.999924 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:39:53.999889    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d\": container with ID starting with d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d not found: ID does not exist" containerID="d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"
Sep 17 19:39:53.999984 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:53.999932    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d"} err="failed to get container status \"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d\": rpc error: code = NotFound desc = could not find container \"d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d\": container with ID starting with d9cd168567fc4185adada72aeed7be0805160c02bd16428a8782bf9d7e640d5d not found: ID does not exist"
Sep 17 19:39:54.006466 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:54.006439    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:39:54.010449 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:54.010424    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:39:55.086453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:55.086407    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" path="/var/lib/kubelet/pods/f1d4be3f-b406-4d4b-aa90-04577f2e36e9/volumes"
Sep 17 19:39:59.737141 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:39:59.737085    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:40:09.737280 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:09.737225    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:40:19.737040 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:19.736980    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:40:29.737040 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:29.736990    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:40:39.746347 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:39.746311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:40:39.755039 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:39.755008    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:40:44.605033 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.604990    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:40:44.605524 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.605489    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main"
Sep 17 19:40:44.605524 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.605505    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main"
Sep 17 19:40:44.605524 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.605522    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="storage-initializer"
Sep 17 19:40:44.605656 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.605531    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="storage-initializer"
Sep 17 19:40:44.605656 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.605615    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d4be3f-b406-4d4b-aa90-04577f2e36e9" containerName="main"
Sep 17 19:40:44.610323 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.610296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.615019 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.614996    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-cr7tr\""
Sep 17 19:40:44.615161 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.615043    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 19:40:44.615161 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.615048    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 19:40:44.615296 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.615167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-ckql9\""
Sep 17 19:40:44.615360 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.615315    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 19:40:44.624196 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.624173    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:40:44.769040 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.769005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2fs\" (UniqueName: \"kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.769248 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.769103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.870383 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.870281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2fs\" (UniqueName: \"kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.870383 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.870358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.873185 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.873161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.879194 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.879167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2fs\" (UniqueName: \"kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:44.922120 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:44.922089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:45.196101 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:45.196071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:40:45.198863 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:40:45.198836    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355127d0_cad1_4dc0_9d7b_0d00fc588b5a.slice/crio-4a13536613d4813e6683c4373711844ac43178151bcde669d55bd6e536b0ef54 WatchSource:0}: Error finding container 4a13536613d4813e6683c4373711844ac43178151bcde669d55bd6e536b0ef54: Status 404 returned error can't find the container with id 4a13536613d4813e6683c4373711844ac43178151bcde669d55bd6e536b0ef54
Sep 17 19:40:45.224650 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:45.224614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" event={"ID":"355127d0-cad1-4dc0-9d7b-0d00fc588b5a","Type":"ContainerStarted","Data":"4a13536613d4813e6683c4373711844ac43178151bcde669d55bd6e536b0ef54"}
Sep 17 19:40:46.231556 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:46.231500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" event={"ID":"355127d0-cad1-4dc0-9d7b-0d00fc588b5a","Type":"ContainerStarted","Data":"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"}
Sep 17 19:40:46.232092 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:46.231815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:40:46.248593 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:40:46.248536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podStartSLOduration=2.248517008 podStartE2EDuration="2.248517008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:40:46.24764718 +0000 UTC m=+3663.389073393" watchObservedRunningTime="2026-09-17 19:40:46.248517008 +0000 UTC m=+3663.389943220"
Sep 17 19:41:02.344466 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:02.344414    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:41:17.238757 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:17.238716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:41:17.371302 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:17.371240    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" containerID="cri-o://21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856" gracePeriod=45
Sep 17 19:41:17.747529 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:17.747482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 19:41:18.133047 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.133019    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:41:18.278394 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278409    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278586    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g75jk\" (UniqueName: \"kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.278889 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache\") pod \"62466d12-173d-452b-a412-c17de257c57a\" (UID: \"62466d12-173d-452b-a412-c17de257c57a\") "
Sep 17 19:41:18.279306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.278887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:18.279306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.279004    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62466d12-173d-452b-a412-c17de257c57a-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.279306 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.279174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache" (OuterVolumeSpecName: "model-cache") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:18.279754 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.279726    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home" (OuterVolumeSpecName: "home") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:18.281253 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.281201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:18.281609 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.281577    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk" (OuterVolumeSpecName: "kube-api-access-g75jk") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "kube-api-access-g75jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:18.281609 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.281603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm" (OuterVolumeSpecName: "dshm") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:18.294603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.294544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:18.302250 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.302228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62466d12-173d-452b-a412-c17de257c57a" (UID: "62466d12-173d-452b-a412-c17de257c57a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:18.379816 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379781    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.379816 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379815    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.380056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379831    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75jk\" (UniqueName: \"kubernetes.io/projected/62466d12-173d-452b-a412-c17de257c57a-kube-api-access-g75jk\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.380056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379843    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62466d12-173d-452b-a412-c17de257c57a-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.380056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379854    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.380056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379865    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.380056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.379876    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62466d12-173d-452b-a412-c17de257c57a-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:41:18.381992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.381573    2561 generic.go:356] "Generic (PLEG): container finished" podID="62466d12-173d-452b-a412-c17de257c57a" containerID="21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856" exitCode=0
Sep 17 19:41:18.381992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.381638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerDied","Data":"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"}
Sep 17 19:41:18.381992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.381692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x" event={"ID":"62466d12-173d-452b-a412-c17de257c57a","Type":"ContainerDied","Data":"6ad5a87ca1fa7d9fd8c5a3658ca84cf4104f42c895a23128b5989f36d463de5e"}
Sep 17 19:41:18.381992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.381722    2561 scope.go:117] "RemoveContainer" containerID="21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"
Sep 17 19:41:18.382325 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.382105    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"
Sep 17 19:41:18.392047 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.392026    2561 scope.go:117] "RemoveContainer" containerID="0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"
Sep 17 19:41:18.401001 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.400978    2561 scope.go:117] "RemoveContainer" containerID="21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"
Sep 17 19:41:18.401314 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:41:18.401293    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856\": container with ID starting with 21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856 not found: ID does not exist" containerID="21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"
Sep 17 19:41:18.401391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.401325    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856"} err="failed to get container status \"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856\": rpc error: code = NotFound desc = could not find container \"21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856\": container with ID starting with 21e4856f96bd060a70114b1b541cdee5be7544fdb83791d4592ddd5cde05b856 not found: ID does not exist"
Sep 17 19:41:18.401391 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.401344    2561 scope.go:117] "RemoveContainer" containerID="0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"
Sep 17 19:41:18.401602 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:41:18.401587    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262\": container with ID starting with 0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262 not found: ID does not exist" containerID="0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"
Sep 17 19:41:18.401652 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.401608    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262"} err="failed to get container status \"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262\": rpc error: code = NotFound desc = could not find container \"0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262\": container with ID starting with 0156f3f3bb4792d067b7ff06a22237c5ffa7cce4dd01f2cb3c70051d34d32262 not found: ID does not exist"
Sep 17 19:41:18.409113 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.409085    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:41:18.417683 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.417651    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-cd94bd5f-nhh6x"]
Sep 17 19:41:18.748553 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:18.747875    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:41:19.086758 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:19.086722    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62466d12-173d-452b-a412-c17de257c57a" path="/var/lib/kubelet/pods/62466d12-173d-452b-a412-c17de257c57a/volumes"
Sep 17 19:41:27.238819 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:27.238780    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:33.771293 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:33.771247    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main" containerID="cri-o://a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364" gracePeriod=45
Sep 17 19:41:37.238740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:37.238692    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:47.238261 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:47.238216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:47.238697 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:47.238322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:41:57.239288 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:41:57.239188    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:42:04.027030 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.026997    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:42:04.204604 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.204569    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs\") pod \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") "
Sep 17 19:42:04.204783 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.204640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm2fs\" (UniqueName: \"kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs\") pod \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\" (UID: \"355127d0-cad1-4dc0-9d7b-0d00fc588b5a\") "
Sep 17 19:42:04.207080 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.207044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "355127d0-cad1-4dc0-9d7b-0d00fc588b5a" (UID: "355127d0-cad1-4dc0-9d7b-0d00fc588b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:42:04.207210 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.207079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs" (OuterVolumeSpecName: "kube-api-access-jm2fs") pod "355127d0-cad1-4dc0-9d7b-0d00fc588b5a" (UID: "355127d0-cad1-4dc0-9d7b-0d00fc588b5a"). InnerVolumeSpecName "kube-api-access-jm2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:42:04.305703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.305665    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:42:04.305703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.305699    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2fs\" (UniqueName: \"kubernetes.io/projected/355127d0-cad1-4dc0-9d7b-0d00fc588b5a-kube-api-access-jm2fs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:42:04.597392 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.597355    2561 generic.go:356] "Generic (PLEG): container finished" podID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerID="a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364" exitCode=0
Sep 17 19:42:04.597570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.597404    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"
Sep 17 19:42:04.597570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.597440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" event={"ID":"355127d0-cad1-4dc0-9d7b-0d00fc588b5a","Type":"ContainerDied","Data":"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"}
Sep 17 19:42:04.597570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.597500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl" event={"ID":"355127d0-cad1-4dc0-9d7b-0d00fc588b5a","Type":"ContainerDied","Data":"4a13536613d4813e6683c4373711844ac43178151bcde669d55bd6e536b0ef54"}
Sep 17 19:42:04.597570 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.597522    2561 scope.go:117] "RemoveContainer" containerID="a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"
Sep 17 19:42:04.607165 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.607146    2561 scope.go:117] "RemoveContainer" containerID="a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"
Sep 17 19:42:04.607452 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:42:04.607426    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364\": container with ID starting with a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364 not found: ID does not exist" containerID="a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"
Sep 17 19:42:04.607511 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.607466    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364"} err="failed to get container status \"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364\": rpc error: code = NotFound desc = could not find container \"a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364\": container with ID starting with a316b1459a59d4c2337efe1965953a251fe45ee4cb85bebc1422a6b6fa01a364 not found: ID does not exist"
Sep 17 19:42:04.621264 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.621240    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:42:04.627918 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:04.627884    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbt6gfl"]
Sep 17 19:42:05.086780 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:05.086749    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" path="/var/lib/kubelet/pods/355127d0-cad1-4dc0-9d7b-0d00fc588b5a/volumes"
Sep 17 19:42:09.803470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803438    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803745    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="storage-initializer"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="storage-initializer"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803773    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803779    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803800    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803846    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62466d12-173d-452b-a412-c17de257c57a" containerName="main"
Sep 17 19:42:09.803864 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.803854    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="355127d0-cad1-4dc0-9d7b-0d00fc588b5a" containerName="main"
Sep 17 19:42:09.808576 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.808551    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.811436 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:42:09.811436 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811404    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fzw5l\""
Sep 17 19:42:09.811704 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811684    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 19:42:09.811836 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811743    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 19:42:09.811836 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811823    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:42:09.811972 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.811835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 19:42:09.822861 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.822838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:42:09.957776 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.957742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.957984 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.957781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7dxv\" (UniqueName: \"kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.957984 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.957820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.957984 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.957882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.957984 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.957975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.958146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.958005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.958146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.958056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:09.958146 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:09.958094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059141 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059141 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059141 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059141 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059480 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059480 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059480 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7dxv\" (UniqueName: \"kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059480 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059695 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059695 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059762 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059804 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.059948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.059927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.061705 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.061678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.062057 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.062032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.071198 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.071175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7dxv\" (UniqueName: \"kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv\") pod \"prestop-hook-test-kserve-b9ddcc467-4rxqj\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.118865 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.118815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:10.403663 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.403632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:42:10.404656 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:42:10.404629    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ac0aad_6bde_4a43_867c_c31877cc9c49.slice/crio-6d5809be0daf7c39bee92d7dcfeeeb1b9751c274d743a276f5eb7c8f6ccd2c30 WatchSource:0}: Error finding container 6d5809be0daf7c39bee92d7dcfeeeb1b9751c274d743a276f5eb7c8f6ccd2c30: Status 404 returned error can't find the container with id 6d5809be0daf7c39bee92d7dcfeeeb1b9751c274d743a276f5eb7c8f6ccd2c30
Sep 17 19:42:10.625713 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.625670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerStarted","Data":"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"}
Sep 17 19:42:10.626018 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:10.625744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerStarted","Data":"6d5809be0daf7c39bee92d7dcfeeeb1b9751c274d743a276f5eb7c8f6ccd2c30"}
Sep 17 19:42:13.648049 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:13.647946    2561 generic.go:356] "Generic (PLEG): container finished" podID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerID="452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594" exitCode=0
Sep 17 19:42:13.648049 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:13.648034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerDied","Data":"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"}
Sep 17 19:42:14.657448 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:14.657405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerStarted","Data":"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"}
Sep 17 19:42:14.686012 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:14.685919    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podStartSLOduration=2.776093061 podStartE2EDuration="5.68587689s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.909783829s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:14.683462961 +0000 UTC m=+3751.824889175" watchObservedRunningTime="2026-09-17 19:42:14.68587689 +0000 UTC m=+3751.827303125"
Sep 17 19:42:15.119831 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:15.119797    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:20.119966 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:20.119929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:42:20.121575 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:20.121535    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:42:26.244470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.244357    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:42:26.246710 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.246685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.249514 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.249488    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 19:42:26.250481 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.250451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-lmv8l\""
Sep 17 19:42:26.250629 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.250529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 19:42:26.250704 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.250645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 19:42:26.250992 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.250967    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-frnng\""
Sep 17 19:42:26.262187 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.262153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:42:26.297504 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.297457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.297504 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.297505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85b4\" (UniqueName: \"kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.398848 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.398805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85b4\" (UniqueName: \"kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.399082 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.398965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.401976 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.401950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.408259 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.408235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85b4\" (UniqueName: \"kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.559100 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.558991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:26.843356 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:26.843313    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:42:27.720603 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:27.720557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" event={"ID":"6db469f0-9ee1-414a-aac5-8ca5b1688732","Type":"ContainerStarted","Data":"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"}
Sep 17 19:42:27.721136 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:27.720668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" event={"ID":"6db469f0-9ee1-414a-aac5-8ca5b1688732","Type":"ContainerStarted","Data":"8ab67d3b834a10fc99a95b62d020571ea73eae50b3a0d9f48a9cc2246ef0b239"}
Sep 17 19:42:27.721136 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:27.720832    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:42:27.740479 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:27.740414    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podStartSLOduration=1.740391502 podStartE2EDuration="1.740391502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:27.739155287 +0000 UTC m=+3764.880581500" watchObservedRunningTime="2026-09-17 19:42:27.740391502 +0000 UTC m=+3764.881817715"
Sep 17 19:42:30.120199 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:30.120148    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:42:40.119849 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:40.119799    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:42:50.119381 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:50.119327    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:42:58.727525 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:42:58.727490    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:43:00.120015 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:00.119968    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:43:00.467356 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:00.467246    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:43:08.727851 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:08.727800    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:43:10.119268 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:10.119218    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:43:15.496500 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:15.496429    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main" containerID="cri-o://39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4" gracePeriod=45
Sep 17 19:43:18.728275 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:18.728239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:43:20.119412 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:20.119362    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:43:28.727977 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:28.727928    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:43:28.728413 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:28.728102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:43:30.119496 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:30.119456    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:43:38.727381 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:38.727340    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:43:40.120260 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:40.120212    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:43:45.752948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.752920    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:43:45.847096 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.847056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s85b4\" (UniqueName: \"kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4\") pod \"6db469f0-9ee1-414a-aac5-8ca5b1688732\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") "
Sep 17 19:43:45.847281 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.847126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs\") pod \"6db469f0-9ee1-414a-aac5-8ca5b1688732\" (UID: \"6db469f0-9ee1-414a-aac5-8ca5b1688732\") "
Sep 17 19:43:45.849657 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.849623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4" (OuterVolumeSpecName: "kube-api-access-s85b4") pod "6db469f0-9ee1-414a-aac5-8ca5b1688732" (UID: "6db469f0-9ee1-414a-aac5-8ca5b1688732"). InnerVolumeSpecName "kube-api-access-s85b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:43:45.849791 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.849657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6db469f0-9ee1-414a-aac5-8ca5b1688732" (UID: "6db469f0-9ee1-414a-aac5-8ca5b1688732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:43:45.948730 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.948692    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85b4\" (UniqueName: \"kubernetes.io/projected/6db469f0-9ee1-414a-aac5-8ca5b1688732-kube-api-access-s85b4\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:43:45.948730 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:45.948727    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db469f0-9ee1-414a-aac5-8ca5b1688732-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:43:46.097933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.097814    2561 generic.go:356] "Generic (PLEG): container finished" podID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerID="39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4" exitCode=0
Sep 17 19:43:46.097933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.097862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"
Sep 17 19:43:46.097933 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.097917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" event={"ID":"6db469f0-9ee1-414a-aac5-8ca5b1688732","Type":"ContainerDied","Data":"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"}
Sep 17 19:43:46.098169 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.097981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858" event={"ID":"6db469f0-9ee1-414a-aac5-8ca5b1688732","Type":"ContainerDied","Data":"8ab67d3b834a10fc99a95b62d020571ea73eae50b3a0d9f48a9cc2246ef0b239"}
Sep 17 19:43:46.098169 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.098006    2561 scope.go:117] "RemoveContainer" containerID="39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"
Sep 17 19:43:46.110548 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.110525    2561 scope.go:117] "RemoveContainer" containerID="39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"
Sep 17 19:43:46.111084 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:43:46.111024    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4\": container with ID starting with 39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4 not found: ID does not exist" containerID="39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"
Sep 17 19:43:46.111084 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.111063    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4"} err="failed to get container status \"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4\": rpc error: code = NotFound desc = could not find container \"39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4\": container with ID starting with 39d6939c4b72cd55aa2bd9981d60d17032c4d53a3194c9f0868250007ab450f4 not found: ID does not exist"
Sep 17 19:43:46.120982 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.120960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:43:46.124814 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:46.124791    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedq858"]
Sep 17 19:43:47.086633 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:47.086601    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" path="/var/lib/kubelet/pods/6db469f0-9ee1-414a-aac5-8ca5b1688732/volumes"
Sep 17 19:43:50.119567 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:43:50.119520    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:44:00.119494 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:00.119440    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:44:10.119948 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:10.119882    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:44:16.275753 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.275713    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:44:16.276251 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.276166    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main"
Sep 17 19:44:16.276251 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.276187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main"
Sep 17 19:44:16.276341 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.276285    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db469f0-9ee1-414a-aac5-8ca5b1688732" containerName="main"
Sep 17 19:44:16.278951 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.278928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.282618 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.282592    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tplz6\""
Sep 17 19:44:16.282742 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.282592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 19:44:16.282742 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.282592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 19:44:16.282742 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.282592    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-v245d\""
Sep 17 19:44:16.283059 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.283043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 19:44:16.287482 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.287460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:44:16.338115 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.338084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.338268 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.338133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49m8\" (UniqueName: \"kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.439283 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.439250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.439454 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.439306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49m8\" (UniqueName: \"kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.442102 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.442075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.447644 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.447620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49m8\" (UniqueName: \"kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.589361 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.589329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:16.877452 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:16.877352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:44:16.881777 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:44:16.881735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f3fa54_ddc5_4137_b0d2_57d13d7e9579.slice/crio-ec9f2cd3300f0b649bc018f5e81cbb13e786b21bf828941c3b82b9aa978c3aef WatchSource:0}: Error finding container ec9f2cd3300f0b649bc018f5e81cbb13e786b21bf828941c3b82b9aa978c3aef: Status 404 returned error can't find the container with id ec9f2cd3300f0b649bc018f5e81cbb13e786b21bf828941c3b82b9aa978c3aef
Sep 17 19:44:17.248888 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:17.248801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" event={"ID":"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579","Type":"ContainerStarted","Data":"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"}
Sep 17 19:44:17.248888 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:17.248857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" event={"ID":"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579","Type":"ContainerStarted","Data":"ec9f2cd3300f0b649bc018f5e81cbb13e786b21bf828941c3b82b9aa978c3aef"}
Sep 17 19:44:17.249101 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:17.248947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:17.295852 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:17.295794    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podStartSLOduration=1.295777364 podStartE2EDuration="1.295777364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:17.294541711 +0000 UTC m=+3874.435967922" watchObservedRunningTime="2026-09-17 19:44:17.295777364 +0000 UTC m=+3874.437203576"
Sep 17 19:44:20.132526 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:20.132492    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:44:20.140178 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:20.140152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:44:23.431878 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:23.431836    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:44:38.454756 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:38.454689    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" containerID="cri-o://da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0" gracePeriod=45
Sep 17 19:44:39.205780 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.205748    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:44:39.342450 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342450 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342505    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342583    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.342703 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.343008 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7dxv\" (UniqueName: \"kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv\") pod \"57ac0aad-6bde-4a43-867c-c31877cc9c49\" (UID: \"57ac0aad-6bde-4a43-867c-c31877cc9c49\") "
Sep 17 19:44:39.343008 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.342807    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache" (OuterVolumeSpecName: "model-cache") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:39.343111 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.343074    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.343638 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.343589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:44:39.343779 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.343754    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home" (OuterVolumeSpecName: "home") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:39.346010 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.345972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm" (OuterVolumeSpecName: "dshm") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:39.347202 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.347168    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:44:39.347685 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.347655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv" (OuterVolumeSpecName: "kube-api-access-h7dxv") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "kube-api-access-h7dxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:44:39.351793 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.351764    2561 generic.go:356] "Generic (PLEG): container finished" podID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerID="da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0" exitCode=0
Sep 17 19:44:39.351912 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.351815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerDied","Data":"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"}
Sep 17 19:44:39.351912 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.351828    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"
Sep 17 19:44:39.351912 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.351869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" event={"ID":"57ac0aad-6bde-4a43-867c-c31877cc9c49","Type":"ContainerDied","Data":"6d5809be0daf7c39bee92d7dcfeeeb1b9751c274d743a276f5eb7c8f6ccd2c30"}
Sep 17 19:44:39.352033 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.351919    2561 scope.go:117] "RemoveContainer" containerID="da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"
Sep 17 19:44:39.362553 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.362526    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:39.368863 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.368814    2561 scope.go:117] "RemoveContainer" containerID="452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"
Sep 17 19:44:39.372830 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.372805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57ac0aad-6bde-4a43-867c-c31877cc9c49" (UID: "57ac0aad-6bde-4a43-867c-c31877cc9c49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:44:39.377543 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.377524    2561 scope.go:117] "RemoveContainer" containerID="da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"
Sep 17 19:44:39.377819 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:44:39.377793    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0\": container with ID starting with da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0 not found: ID does not exist" containerID="da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"
Sep 17 19:44:39.377875 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.377821    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0"} err="failed to get container status \"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0\": rpc error: code = NotFound desc = could not find container \"da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0\": container with ID starting with da05f81fb15407f6e009bfdccac5fc10e90624605548521c5df92f449ad61eb0 not found: ID does not exist"
Sep 17 19:44:39.377875 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.377839    2561 scope.go:117] "RemoveContainer" containerID="452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"
Sep 17 19:44:39.378113 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:44:39.378087    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594\": container with ID starting with 452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594 not found: ID does not exist" containerID="452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"
Sep 17 19:44:39.378186 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.378123    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594"} err="failed to get container status \"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594\": rpc error: code = NotFound desc = could not find container \"452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594\": container with ID starting with 452769a96ec5e30604851a7c67300b360537a0a95886d0b4fbf9f76f60d22594 not found: ID does not exist"
Sep 17 19:44:39.444016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.443991    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444016 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444012    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444021    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444030    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ac0aad-6bde-4a43-867c-c31877cc9c49-cabundle-cert\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444038    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ac0aad-6bde-4a43-867c-c31877cc9c49-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444046    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7dxv\" (UniqueName: \"kubernetes.io/projected/57ac0aad-6bde-4a43-867c-c31877cc9c49-kube-api-access-h7dxv\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.444154 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.444056    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ac0aad-6bde-4a43-867c-c31877cc9c49-kserve-provision-location\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:44:39.675653 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.675616    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:44:39.683449 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:39.683423    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj"]
Sep 17 19:44:40.133990 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:40.133949    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-4rxqj" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Sep 17 19:44:41.086254 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:41.086217    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" path="/var/lib/kubelet/pods/57ac0aad-6bde-4a43-867c-c31877cc9c49/volumes"
Sep 17 19:44:45.305299 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:45.305269    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:44:45.308025 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:45.308003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:44:48.254538 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:48.254511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:44:50.097195 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:50.097163    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:44:58.255249 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:44:58.255205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:05.117251 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:05.117206    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main" containerID="cri-o://2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85" gracePeriod=45
Sep 17 19:45:08.254810 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:08.254768    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:18.255197 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:18.255159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:18.255640 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:18.255257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:45:28.254953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:28.254840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:35.368121 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.368096    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:45:35.519252 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.519172    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t49m8\" (UniqueName: \"kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8\") pod \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") "
Sep 17 19:45:35.519252 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.519207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs\") pod \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\" (UID: \"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579\") "
Sep 17 19:45:35.521659 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.521623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" (UID: "d8f3fa54-ddc5-4137-b0d2-57d13d7e9579"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:45:35.521772 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.521674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8" (OuterVolumeSpecName: "kube-api-access-t49m8") pod "d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" (UID: "d8f3fa54-ddc5-4137-b0d2-57d13d7e9579"). InnerVolumeSpecName "kube-api-access-t49m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:45:35.611631 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.611599    2561 generic.go:356] "Generic (PLEG): container finished" podID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerID="2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85" exitCode=0
Sep 17 19:45:35.611752 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.611650    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"
Sep 17 19:45:35.611752 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.611648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" event={"ID":"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579","Type":"ContainerDied","Data":"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"}
Sep 17 19:45:35.611830 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.611779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj" event={"ID":"d8f3fa54-ddc5-4137-b0d2-57d13d7e9579","Type":"ContainerDied","Data":"ec9f2cd3300f0b649bc018f5e81cbb13e786b21bf828941c3b82b9aa978c3aef"}
Sep 17 19:45:35.611830 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.611813    2561 scope.go:117] "RemoveContainer" containerID="2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"
Sep 17 19:45:35.619789 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.619766    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49m8\" (UniqueName: \"kubernetes.io/projected/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-kube-api-access-t49m8\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:45:35.619789 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.619790    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:45:35.620482 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.620464    2561 scope.go:117] "RemoveContainer" containerID="2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"
Sep 17 19:45:35.620763 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:45:35.620745    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85\": container with ID starting with 2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85 not found: ID does not exist" containerID="2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"
Sep 17 19:45:35.620810 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.620772    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85"} err="failed to get container status \"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85\": rpc error: code = NotFound desc = could not find container \"2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85\": container with ID starting with 2d084699218bc60d8ed1fa8d2f9d262339e67841204ba691e2226876b4dc2a85 not found: ID does not exist"
Sep 17 19:45:35.632754 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.632733    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:45:35.636255 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:35.636232    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98kjsj"]
Sep 17 19:45:37.086657 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:45:37.086620    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" path="/var/lib/kubelet/pods/d8f3fa54-ddc5-4137-b0d2-57d13d7e9579/volumes"
Sep 17 19:46:18.193881 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.193844    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194182    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194195    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194211    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194217    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194234    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="storage-initializer"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194242    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="storage-initializer"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194294    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f3fa54-ddc5-4137-b0d2-57d13d7e9579" containerName="main"
Sep 17 19:46:18.194338 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.194300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ac0aad-6bde-4a43-867c-c31877cc9c49" containerName="main"
Sep 17 19:46:18.197447 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.197429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.200806 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.200780    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:46:18.200998 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.200888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:46:18.200998 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.200926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-dx5j7\""
Sep 17 19:46:18.201127 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.200888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:46:18.207886 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.207862    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:18.276049 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.276208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.276208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.276208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.276208 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5tf\" (UniqueName: \"kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.276363 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.276211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377328 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377563 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377563 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377563 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5tf\" (UniqueName: \"kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377563 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377787 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.377855 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.377808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.379920 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.379881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.380225 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.380205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.386633 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.386588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5tf\" (UniqueName: \"kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf\") pod \"rolling-upgrade-test-kserve-74b79d7f77-bj8nh\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.509453 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.509363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:18.820597 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:18.819982    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:18.822543 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:46:18.822505    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219ad0e7_3e55_445b_8b86_2f4893268425.slice/crio-f34d3ca7d049c5ced832b416077c91c99510eb12f814bd25c71edbe64e095461 WatchSource:0}: Error finding container f34d3ca7d049c5ced832b416077c91c99510eb12f814bd25c71edbe64e095461: Status 404 returned error can't find the container with id f34d3ca7d049c5ced832b416077c91c99510eb12f814bd25c71edbe64e095461
Sep 17 19:46:19.806664 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:19.806630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" event={"ID":"219ad0e7-3e55-445b-8b86-2f4893268425","Type":"ContainerStarted","Data":"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"}
Sep 17 19:46:19.807111 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:19.806681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" event={"ID":"219ad0e7-3e55-445b-8b86-2f4893268425","Type":"ContainerStarted","Data":"f34d3ca7d049c5ced832b416077c91c99510eb12f814bd25c71edbe64e095461"}
Sep 17 19:46:20.510244 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:20.510206    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:28.510558 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:28.510501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:28.523180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:28.523156    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:28.541964 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:28.541885    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" podStartSLOduration=10.541870316 podStartE2EDuration="10.541870316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:19.824045276 +0000 UTC m=+3996.965471489" watchObservedRunningTime="2026-09-17 19:46:28.541870316 +0000 UTC m=+4005.683296591"
Sep 17 19:46:28.874524 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:28.874494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:55.329694 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.329600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:55.339916 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:46:55.339862    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:46:55.339916 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:55.339916 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:55.339916 ip-10-0-136-146 kubenswrapper[2561]:  > containerID="54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2" cmd=["/bin/sleep","15"]
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.339938    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" message="" err=<
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:  >
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:46:55.339952    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" containerName="main" containerID="cri-o://54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"
Sep 17 19:46:55.340187 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.340018    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" containerName="main" containerID="cri-o://54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2" gracePeriod=60
Sep 17 19:46:55.686490 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.686464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:55.808754 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.808937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv5tf\" (UniqueName: \"kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.808937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808830    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.808937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.808937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.808937 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.808923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs\") pod \"219ad0e7-3e55-445b-8b86-2f4893268425\" (UID: \"219ad0e7-3e55-445b-8b86-2f4893268425\") "
Sep 17 19:46:55.809222 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.809153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:55.809222 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.809169    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home" (OuterVolumeSpecName: "home") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:55.809298 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.809221    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache" (OuterVolumeSpecName: "model-cache") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:55.811301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.811268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm" (OuterVolumeSpecName: "dshm") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:55.811301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.811281    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf" (OuterVolumeSpecName: "kube-api-access-xv5tf") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "kube-api-access-xv5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:46:55.811438 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.811306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "219ad0e7-3e55-445b-8b86-2f4893268425" (UID: "219ad0e7-3e55-445b-8b86-2f4893268425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909831    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5tf\" (UniqueName: \"kubernetes.io/projected/219ad0e7-3e55-445b-8b86-2f4893268425-kube-api-access-xv5tf\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909855    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-tmp-dir\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909867    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-home\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909876    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-model-cache\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909883    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219ad0e7-3e55-445b-8b86-2f4893268425-tls-certs\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.909922 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.909890    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219ad0e7-3e55-445b-8b86-2f4893268425-dshm\") on node \"ip-10-0-136-146.ec2.internal\" DevicePath \"\""
Sep 17 19:46:55.985878 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.985837    2561 generic.go:356] "Generic (PLEG): container finished" podID="219ad0e7-3e55-445b-8b86-2f4893268425" containerID="54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2" exitCode=0
Sep 17 19:46:55.986056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.985939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" event={"ID":"219ad0e7-3e55-445b-8b86-2f4893268425","Type":"ContainerDied","Data":"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"}
Sep 17 19:46:55.986056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.985958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"
Sep 17 19:46:55.986056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.986006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh" event={"ID":"219ad0e7-3e55-445b-8b86-2f4893268425","Type":"ContainerDied","Data":"f34d3ca7d049c5ced832b416077c91c99510eb12f814bd25c71edbe64e095461"}
Sep 17 19:46:55.986056 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.986050    2561 scope.go:117] "RemoveContainer" containerID="54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"
Sep 17 19:46:55.997654 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.997636    2561 scope.go:117] "RemoveContainer" containerID="54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"
Sep 17 19:46:55.997913 ip-10-0-136-146 kubenswrapper[2561]: E0917 19:46:55.997875    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2\": container with ID starting with 54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2 not found: ID does not exist" containerID="54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"
Sep 17 19:46:55.997981 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:55.997915    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2"} err="failed to get container status \"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2\": rpc error: code = NotFound desc = could not find container \"54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2\": container with ID starting with 54f163543b1c897e804141c9df4ed5342eb29ea970067b36e4cb0bf3024ebff2 not found: ID does not exist"
Sep 17 19:46:56.008784 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:56.008756    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:56.012774 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:56.012752    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-bj8nh"]
Sep 17 19:46:57.086384 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:46:57.086350    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" path="/var/lib/kubelet/pods/219ad0e7-3e55-445b-8b86-2f4893268425/volumes"
Sep 17 19:48:08.994127 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:08.994095    2561 ???:1] "http: TLS handshake error from 10.0.129.204:47568: EOF"
Sep 17 19:48:09.002399 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:09.002368    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/0.log"
Sep 17 19:48:09.007369 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:09.007349    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/1.log"
Sep 17 19:48:09.817287 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:09.817244    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/0.log"
Sep 17 19:48:09.822147 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:09.822120    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/1.log"
Sep 17 19:48:10.798653 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:10.798618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gxlc4_3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74/authorino/0.log"
Sep 17 19:48:10.989224 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:10.989189    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dlnhg_24e2a03d-efdf-4aed-ba4f-b634ff5d050f/manager/0.log"
Sep 17 19:48:19.764029 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:19.764000    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzhx6_1b8b0027-07c0-4dde-b7af-6418d620ef89/global-pull-secret-syncer/0.log"
Sep 17 19:48:19.894971 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:19.894942    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v6jfk_63335224-a809-468b-9c26-86a7c3ba6a77/konnectivity-agent/0.log"
Sep 17 19:48:19.985695 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:19.985665    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-146.ec2.internal_056c7b5706208129dfc0377a36f97f3c/haproxy/0.log"
Sep 17 19:48:24.601404 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:24.601308    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gxlc4_3ddfb2f2-2d9b-4dbf-8083-7b11729e6e74/authorino/0.log"
Sep 17 19:48:24.889369 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:24.889273    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dlnhg_24e2a03d-efdf-4aed-ba4f-b634ff5d050f/manager/0.log"
Sep 17 19:48:26.283378 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.283291    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/alertmanager/0.log"
Sep 17 19:48:26.305360 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.305325    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/config-reloader/0.log"
Sep 17 19:48:26.328783 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.328765    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/kube-rbac-proxy-web/0.log"
Sep 17 19:48:26.351793 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.351778    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/kube-rbac-proxy/0.log"
Sep 17 19:48:26.375443 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.375416    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/kube-rbac-proxy-metric/0.log"
Sep 17 19:48:26.398172 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.398151    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/prom-label-proxy/0.log"
Sep 17 19:48:26.419350 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.419329    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad880bff-d546-4361-a1f0-594ec9783d3e/init-config-reloader/0.log"
Sep 17 19:48:26.584470 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.584444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6589774446-4gl4j_42c40e78-8f7a-4dc3-b22e-05d13f41d58c/metrics-server/0.log"
Sep 17 19:48:26.613713 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.613690    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-jm7w9_6e78f413-dd78-415b-bce5-d90bdbaeae37/monitoring-plugin/0.log"
Sep 17 19:48:26.643180 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.643155    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d4xc_4a608575-01b3-40bd-b73a-ee9b1ba68c9e/node-exporter/0.log"
Sep 17 19:48:26.670854 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.670830    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d4xc_4a608575-01b3-40bd-b73a-ee9b1ba68c9e/kube-rbac-proxy/0.log"
Sep 17 19:48:26.694807 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:26.694790    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d4xc_4a608575-01b3-40bd-b73a-ee9b1ba68c9e/init-textfile/0.log"
Sep 17 19:48:27.020693 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.020665    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/prometheus/0.log"
Sep 17 19:48:27.039663 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.039642    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/config-reloader/0.log"
Sep 17 19:48:27.062950 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.062926    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/thanos-sidecar/0.log"
Sep 17 19:48:27.092262 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.092234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/kube-rbac-proxy-web/0.log"
Sep 17 19:48:27.113701 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.113685    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/kube-rbac-proxy/0.log"
Sep 17 19:48:27.135249 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.135223    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/kube-rbac-proxy-thanos/0.log"
Sep 17 19:48:27.157037 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.157018    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e13aa3db-d679-465d-8a56-ec68b30f70bc/init-config-reloader/0.log"
Sep 17 19:48:27.188020 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.188001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-t8ms6_7d8e8d47-9129-47d3-8d46-9ca191d6a55c/prometheus-operator/0.log"
Sep 17 19:48:27.204513 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:27.204492    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-t8ms6_7d8e8d47-9129-47d3-8d46-9ca191d6a55c/kube-rbac-proxy/0.log"
Sep 17 19:48:30.406602 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.406566    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"]
Sep 17 19:48:30.407129 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.407072    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" containerName="main"
Sep 17 19:48:30.407129 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.407089    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" containerName="main"
Sep 17 19:48:30.407207 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.407152    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="219ad0e7-3e55-445b-8b86-2f4893268425" containerName="main"
Sep 17 19:48:30.410345 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.410326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.413188 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.413167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7vd6\"/\"default-dockercfg-z24dx\""
Sep 17 19:48:30.413478 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.413460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"openshift-service-ca.crt\""
Sep 17 19:48:30.414133 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.414117    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"kube-root-ca.crt\""
Sep 17 19:48:30.421750 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.421724    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"]
Sep 17 19:48:30.470181 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.470158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-podres\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.470301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.470220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-proc\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.470301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.470256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-lib-modules\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.470301 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.470273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-sys\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.470414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.470306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dwp\" (UniqueName: \"kubernetes.io/projected/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-kube-api-access-s9dwp\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571474 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-podres\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-proc\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-lib-modules\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-sys\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-podres\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dwp\" (UniqueName: \"kubernetes.io/projected/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-kube-api-access-s9dwp\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571667 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-proc\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571943 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-sys\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.571943 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.571688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-lib-modules\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.582233 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.582211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dwp\" (UniqueName: \"kubernetes.io/projected/17bd7aa7-b386-4fb4-a87d-eca330d05ce9-kube-api-access-s9dwp\") pod \"perf-node-gather-daemonset-g6g7r\" (UID: \"17bd7aa7-b386-4fb4-a87d-eca330d05ce9\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.720358 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.720279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:30.992080 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:30.992052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"]
Sep 17 19:48:30.994678 ip-10-0-136-146 kubenswrapper[2561]: W0917 19:48:30.994650    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17bd7aa7_b386_4fb4_a87d_eca330d05ce9.slice/crio-ad33d30c960586397311c494b841a51a28ba1df22f1caebd9616ee51e72b8a2f WatchSource:0}: Error finding container ad33d30c960586397311c494b841a51a28ba1df22f1caebd9616ee51e72b8a2f: Status 404 returned error can't find the container with id ad33d30c960586397311c494b841a51a28ba1df22f1caebd9616ee51e72b8a2f
Sep 17 19:48:31.392054 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.392005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r" event={"ID":"17bd7aa7-b386-4fb4-a87d-eca330d05ce9","Type":"ContainerStarted","Data":"bf6d1491d332a93fe49a1b2ab4d304b319e0fd1da5761a45152280d29afcafd7"}
Sep 17 19:48:31.392234 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.392074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r" event={"ID":"17bd7aa7-b386-4fb4-a87d-eca330d05ce9","Type":"ContainerStarted","Data":"ad33d30c960586397311c494b841a51a28ba1df22f1caebd9616ee51e72b8a2f"}
Sep 17 19:48:31.392234 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.392185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:31.411380 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.411322    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r" podStartSLOduration=1.41130761 podStartE2EDuration="1.41130761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:48:31.409681069 +0000 UTC m=+4128.551107283" watchObservedRunningTime="2026-09-17 19:48:31.41130761 +0000 UTC m=+4128.552733822"
Sep 17 19:48:31.799825 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.799739    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q88zg_0a78cc5a-39bd-4867-921b-dee7019942e6/dns/0.log"
Sep 17 19:48:31.819064 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.819038    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q88zg_0a78cc5a-39bd-4867-921b-dee7019942e6/kube-rbac-proxy/0.log"
Sep 17 19:48:31.936833 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:31.936806    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vqtgn_dd2216fe-b76c-4e0c-aed3-0533173eacb1/dns-node-resolver/0.log"
Sep 17 19:48:32.480746 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:32.480709    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74bc5b974-xsm24_5e1253ab-f0d2-407d-b53e-2ab940f3317a/registry/0.log"
Sep 17 19:48:32.559974 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:32.559946    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p7tld_64e79258-76fa-4c91-922a-a43748ad79ac/node-ca/0.log"
Sep 17 19:48:33.582715 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:33.582682    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/0.log"
Sep 17 19:48:33.587426 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:33.587399    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/1.log"
Sep 17 19:48:34.194639 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:34.194609    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-npxf9_54482567-ff8b-4b5b-a824-e6dbf27fd9dd/serve-healthcheck-canary/0.log"
Sep 17 19:48:34.684137 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:34.684099    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z5j4_c947bcc9-94d2-4709-a322-8547e2a9ccb2/kube-rbac-proxy/0.log"
Sep 17 19:48:34.726740 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:34.726717    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z5j4_c947bcc9-94d2-4709-a322-8547e2a9ccb2/exporter/0.log"
Sep 17 19:48:34.773472 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:34.773440    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2z5j4_c947bcc9-94d2-4709-a322-8547e2a9ccb2/extractor/0.log"
Sep 17 19:48:37.407627 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:37.407593    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-g6g7r"
Sep 17 19:48:38.023265 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:38.023239    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54799454dd-875gt_14a7b116-59d4-4e29-978c-a2a450a401f0/manager/0.log"
Sep 17 19:48:39.240545 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:39.240493    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b569b69c4-pd6vx_0b1c61f4-72fa-4fd0-90e4-e31a6c27cebb/manager/0.log"
Sep 17 19:48:40.357345 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:40.357305    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/0.log"
Sep 17 19:48:40.364431 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:40.364352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/1.log"
Sep 17 19:48:41.541953 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:41.541884    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/0.log"
Sep 17 19:48:41.546812 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:41.546790    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-lzj5k_5a413b68-67ab-4c5f-a450-cd459e093103/istio-proxy/1.log"
Sep 17 19:48:46.618198 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.618159    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:48:46.637827 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.637794    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/egress-router-binary-copy/0.log"
Sep 17 19:48:46.658348 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.658326    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/cni-plugins/0.log"
Sep 17 19:48:46.679610 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.679586    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/bond-cni-plugin/0.log"
Sep 17 19:48:46.699115 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.699098    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/routeoverride-cni/0.log"
Sep 17 19:48:46.717218 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.717199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/whereabouts-cni-bincopy/0.log"
Sep 17 19:48:46.735755 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.735733    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkmb7_15e4ce2c-6fb3-4ad9-a073-f50607348e91/whereabouts-cni/0.log"
Sep 17 19:48:46.965786 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:46.965703    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qqkfr_8e79dc1c-f159-4707-8051-2b4b315d1048/kube-multus/0.log"
Sep 17 19:48:47.062343 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:47.062315    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b9cf4_05ff87bb-d370-4f09-921b-a00049948b21/network-metrics-daemon/0.log"
Sep 17 19:48:47.099624 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:47.099593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b9cf4_05ff87bb-d370-4f09-921b-a00049948b21/kube-rbac-proxy/0.log"
Sep 17 19:48:47.995810 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:47.995780    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-controller/0.log"
Sep 17 19:48:48.011414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.011389    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/0.log"
Sep 17 19:48:48.011747 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.011724    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovn-acl-logging/1.log"
Sep 17 19:48:48.028866 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.028846    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/kube-rbac-proxy-node/0.log"
Sep 17 19:48:48.049174 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.049150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:48:48.068744 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.068719    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/northd/0.log"
Sep 17 19:48:48.088176 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.088136    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/nbdb/0.log"
Sep 17 19:48:48.107414 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.107376    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/sbdb/0.log"
Sep 17 19:48:48.187723 ip-10-0-136-146 kubenswrapper[2561]: I0917 19:48:48.187697    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-22lq4_26c17490-9136-4f0e-8e2f-ade74d14bdd6/ovnkube-controller/0.log"